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FOREWORD 


Improvements  in  Army  training  and  evaluation  are  an  enduring  concern  of  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI).  A  related  concern  addressed  in 
this  effort  is  that  military  officers  may  have  insufficient  opportunity  to  apply  their  battle  command 
reasoning  skills  in  realistic  battle  command  situations.  One  way  to  provide  sufficient  opportunity 
is  through  an  intelligent  tutoring  system  (ITS)  for  training  battlefield  command  reasoning  skills. 

This  Phase  I  Small  Business  Innovation  Research  Program  (SBIR)  effort  targeted  the 
design  and  limited  demonstration  of  a  Socratic  ITS  for  high-level  battlefield  command  reasoning 
skills.  The  research  goal  was  to  develop  innovative  training  methods  for  conceptual  skills, 
particularly  new  ITS  techniques  and  technology  for  teaching  skills  that  cannot  be  taught  as  simple 
methods  and  procedures  to  be  followed.  The  product  goal  was  an  ITS  for  training  battlefield 
command  reasoning  skills  that  could  be  hosted  on  an  interactive  web  site.  Overall,  the  objectives 
for  developing  such  an  ITS  prototype  include:  anytime,  anywhere  tutoring;  deliberate  practice 
opportunities;  standardized  instructional  procedures;  and  at  least  a  partial  answer  to  the  growing 
problem  of  limited  expert  human  tutors. 

This  research  was  part  of  ARI’s  Future  Battlefield  Conditions  (FBC)  team  efforts  to 
enhance  soldier  preparedness  through  development  of  training  and  evaluation  methods  to  meet 
future  battlefield  conditions.  This  report  represents  efforts  for  Work  Package  211,  Techniques  and 
Tools  for  Command,  Control,  Communications,  Computer,  Intelligence,  Surveillance,  and 
Reconnaissance  (C4ISR)  Training  of  Future  Brigade  Combat  Team  Commanders  and  Staffs 
(FUTURETRAIN).  Results  of  this  effort  were  briefed  to  the  Director  of  Bio-Systems,  Office  of 
the  Director,  Defense  Research  and  Engineering.  As  a  result  of  the  Phase  I  success,  the  Phase  II 
effort  was  awarded  and  an  ITS  for  conceptual  skill  training  should  be  available  for  commercial 
application  by  September  2003. 


MICHAEL  G.  RUMSEY~~ 
Acting  Technical  Director 
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PHASE  I  FINAL  REPORT  ON  AN  INTELLIGENT  TUTORING  SYSTEM  FOR  TEACHING 
BATTLEFIELD  COMMAND  REASONING  SKILLS 


EXECUTIVE  SUMMARY _ 

Research  Requirement: 

Under  extreme  stress,  time  pressure,  and  uncertainty,  commanders  must  assess  complex, 
ambiguous  tactical  situations,  develop  and  prioritize  goals,  and  pursue  these  goals  by  issuing 
orders,  reports,  and  requests  that  consider  the  situation,  own  assets  and  capabilities,  and  the  assets, 
capabilities,  and  intentions  of  the  enemy.  High  performance  decision-making  requires  the 
commander  to  apply  detailed  and  situation-specific  knowledge  as  well  as  high-level  thinking 
habits  and  skills  that  are  applicable  across  diverse  tactical  situations.  These  skills  include  modeling 
a  thinking  enemy,  using  all  available  assets,  and  considering  how  the  commander's  fight  fits  into 
the  bigger  picture  from  friendly  and  enemy  perspectives. 

These  reasoning  skills  cannot  be  taught  simply  as  methods  and  procedures  to  be  followed. 
Although  guidelines  can  help  commanders  achieve  baseline  levels  of  performance,  achieving  an 
expert  level  of  proficiency  requires: 

•  Extensive  practice  of  command  reasoning  skills  in  a  variety  of  situations  to  acquire  and 
reinforce  skills  until  they  can  be  applied  flexibly  and  automatically.  This  experience  can 
include  a  combination  of  actual  combat,  live  training  exercises,  computer-based  and  paper- 
based  simulations,  and  mental  exercises. 

•  Coaching  and  feedback  from  instructors  to  identify  strengths  and  weaknesses  in 
performance,  encourage  reflection,  stimulate  the  student's  thought  processes,  and  enhance 
the  student's  reasoning  capabilities. 

Coaching  and  feedback  are  typically  provided  by  expert  instructors  during  dialogs  with 
students.  An  example  of  this  type  of  instruction  includes  Think  Like  a  Commander  (TLAC) 
materials  (U.S.  Army  Research  Institute,  2001).  These  TLAC  materials  are  currently  used  in 
officer  courses  at  Fort  Leavenworth  and  Fort  Knox.  Another  example  is  Tactical  Decision  Games 
(TDGs)  (e.g.,  Schmitt,  1994).  In  both  these  forms  of  training,  tactical  scenarios  are  presented  to 
learners  who  must  then  reach  conclusions  and  make  decisions.  If  the  learner  fails  to  take  an 
applicable  theme  into  account,  the  instructor  asks  the  student  questions,  at  first  indirect,  to 
encourage  the  student  to  reflect  upon  his  or  her  thinking  about  a  theme. 

An  inherent  limitation  of  mentored  instruction  is  that  the  expert  mentors  are  an  expensive 
limited  resource  that  is  infrequently  available;  when  available  their  attention  must  often  be  shared 
among  too  many  learners  for  truly  effective  personalized  interaction.  Classroom  instruction,  such 
as  Army  officer  training,  adds  the  constraint  that  instruction  is  only  accessible  to  learners  who  can 
attend  the  course  on  site,  requiring  significant  time  and  travel  costs  as  well  as  scheduling 
difficulties.  Automated,  computer-based  training  can  provide  instruction  “any  time,  any  place.” 
However,  the  types  of  expert  coaching  and  mentoring  required  to  teach  these  command  reasoning 
skills  require  far  more  sophisticated  reasoning  than  is  possible  with  current  computer-based 
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training  (CBT)  and  web-based  training  (WBT)  technologies  that  typically  assess  learners  based  on 
simple  multiple-choice  and  fill-in-the-blank  questions. 

Intelligent  Tutoring  Systems  (ITSs)  are  computer-based  training  systems  that  mimic  human 
instructors  to  provide  automated,  one-on-one  instruction.  Intelligent  Tutoring  Systems  encode 
subject  matter  and  instructional  expertise  to  assess  each  learner's  performance,  knowledge,  and 
skills  to  provide  individualized  learning  experiences,  adapted  to  each  learner's  skills,  background, 
and  learning  preferences.  Most  ITSs  provide  a  simulation  or  problem-solving  user  interface  that 
enables  learners  to  solve  problems  or  pursue  goals  by  entering  a  sequence  of  actions  or  decisions 
selected  from  a  wide  range  of  possible  choices.  The  ITS  applies  subject  matter  expertise  to 
evaluate  these  actions  and  decisions  to  assess  the  learner's  knowledge  and  skills  and  then  select 
and  deliver  appropriate  instructional  interventions. 

However,  no  ITS  system  developed  to  date  supports  a  domain  nearly  as  complex  as 
battlefield  command  which  requires  sophisticated  reasoning  about  a  large  number  of  goals, 
potential  plans,  variables,  and  constraints.  Evaluation  of  each  learner's  solution  is  extremely 
challenging  because  there  is  no  single  "correct"  solution  to  each  scenario.  As  Gen.  George  S. 

Patton  wrote,  “There  is  no  approved  solution  to  any  tactical  situation.”  Thus,  developing  an  ITS 
for  teaching  battlefield  command  reasoning  skills  will  require  "pushing  the  envelope"  in  ITS 
technology  along  a  number  of  dimensions,  such  as: 

•  Representing  solutions  and  their  rationale  to  complex  battlefield  command  problems, 

•  Assessing  the  learner's  reasoning  skills  by  comparing  each  learner's  solution  with  multiple 

expert  solutions  and  their  rationale,  and 

•  Implementing  learner-mentor  interactions  that  enhance  high-level  thinking  habits,  based  on 

an  assessment  of  each  learner's  reasoning,  as  well  as  knowledge  structures  that  represent 

each  expert's  reasoning. 

Procedure: 

During  Phase  I,  our  objectives  were  to: 

1 .  Identify  the  kinds  of  tactical  analysis  knowledge  that  should  be  represented  by  the 
system  to  support  the  evaluation  of  learner  solutions  to  tactical  scenarios,  in  order  to  refine  the 
system's  student  model. 

2.  Explore  and  evaluate  candidate  interaction  techniques  that  can  achieve  the  types  of 
instmctional  goals  currently  achieved  in  leamer/mentor  interactions  with  expert  human  instructors. 

3.  Identify  the  types  of  automated  tutoring  strategies  needed  to  select  and  generate 
effective  instructional  interventions  based  on  a  student  model. 

4.  Develop  methods  for  authoring  ITS  scenarios  by  entering  good  and  bad  sets  of 
decisions  and  rationale,  and  then  annotating  these  solutions  with  skills  that  are  demonstrated 
(positively  or  negatively)  by  each  solution. 
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5.  Develop  a  limited,  proof-of-concept  software  prototype  that  illustrates  key  ideas 
related  to  leamer/mentor  interaction  methods,  tactical  analysis  knowledge  representation,  and 
interactive  tutoring  strategies. 

Our  core  activities  during  the  Phase  I  effort  included: 

a.  Study  of  prior  work  related  to  the  project  goals  (e.g.,  reasoning  skills  training,  ITS 
techniques  and  Socratic  Tutoring,  and  the  existing  Think  Like  a  Commander  course). 

b.  Conduct,  observation,  and  analysis  of  a  series  of  Tactical  Decision  Game  tutoring 
sessions  between  active-duty  Marine  Captains  and  acknowledged  tactical  (and  tutoring)  experts. 

c.  Design  and  construction  of  a  proof-of-concept  Command  Mentoring  Intelligent  Tutoring 
System  (ComMentor)  prototype  based  on  a  single  TDG  scenario  covered  in  multiple  tutoring 
sessions. 

With  respect  to  our  original  objectives,  activities  ‘a’  and  ‘b’  directly  addressed  objectives  1, 
2,  and  3.  Our  early  work  on  activity  ‘a’  helped  frame  the  problem.  The  extensive  observations 
and  analysis  in  activity  ‘b’  ended  up  forming  the  primary  basis  for  most  of  our  work;  identification 
of  knowledge  form  and  content,  interaction  techniques,  and  tutoring  strategies  all  fed  forward  into 
later  objectives.  Activity  ‘c’  directly  addressed  objectives  4  and  5.  Our  work  on  objective  4  was 
not  only  a  necessary  precondition  to  success  on  objective  5,  but  forms  the  groundwork  for  Phase  II 
work  on  task-specific  authoring  tools.  Likewise  the  prototype  produced  for  objective  5  not  only 
demonstrates  the  feasibility  of  our  ideas,  but  also  serves  as  a  design  base  and  source  of  lessons 
learned  for  our  proposed  further  system  development  during  Phase  II. 

Findings: 

Our  major  Phase  I  accomplishments  fall  into  four  categories: 

1 .  Developing  a  deep  understanding  of  the  battlefield  command  domain,  and  of  existing 
techniques  for  training  battlefield  command  reasoning  skills  (see  “Findings  Based  on  ARI 
Direction  and  Materials”). 

2.  Observing,  analyzing,  and  classifying  Socratic  tutoring  techniques  for  battlefield 
command  reasoning  skills  (see  “Findings  Based  on  TDG  Observations  and  Analysis”). 

3.  Designing  and  building  a  limited  proof-of-concept  prototype  system  called  ComMentor 
that  demonstrates  an  automated  implementation  of  many  tutoring  techniques  observed  and 
cataloged  above  (see  “PHASE  I  PROTOTYPE  SYSTEM”). 

4.  Developing  a  system  design  and  work  plan  for  the  Phase  II  implementation  of  a  full 
ComMentor  prototype  (see  “PHASE  II  SYSTEM  DESIGN”  and  “FUTURE  WORK”). 


IX 


Utilization  of  Findings: 

In  this  Phase  I  project,  we  have  developed  and  documented  a  detailed  understanding  of 
how  expert  human  tutors  Socratically  guide  junior  commanders  through  training  scenarios  in  order 
to  drill  and  improve  their  battlefield  command  reasoning  skills.  Further,  we  have  demonstrated, 
through  construction  of  a  limited  proof-of-concept  prototype,  that  many  of  the  observed  tutoring 
behaviors  can  be  duplicated  by  an  ITS.  Finally,  we  have  developed  a  detailed  plan  for  how  to 
proceed  with  Phase  II  development  of  a  complete,  evaluable,  ComMentor  prototype.  As  the  Phase 
II  proposal  was  awarded,  the  findings  will  be  used  to  complete  development  of  a  prototype  ITS  for 
battle  command  reasoning  skills  by  September  2003. 
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PHASE  I  FINAL  REPORT  ON  AN  INTELLIGENT  TUTORING  SYSTEM  FOR  TEACHING 
BATTLEFIELD  COMMAND  REASONING  SKILLS 


Introduction 


Overview  of  Problem  and  Solution 

Under  extreme  stress,  time  pressure,  and  uncertainty,  commanders  must  assess  complex, 
ambiguous  tactical  situations,  develop  and  prioritize  goals,  and  pursue  these  goals  by  issuing 
orders,  reports,  and  requests  that  consider  the  situation,  own  assets  and  capabilities,  and  the 
assets,  capabilities,  and  intentions  of  the  enemy.  High  performance  decision-making  requires  the 
commander  to  apply  detailed  and  situation-specific  knowledge  as  well  as  high-level  thinking 
habits  and  skills  that  are  applicable  across  diverse  tactical  situations.  These  skills  include 
modeling  a  thinking  enemy,  using  all  available  assets,  and  considering  how  the  commander's 
fight  Jits  into  the  bigger  picture  from  friendly  and  enemy  perspectives. 

Problem.  These  reasoning  skills  cannot  be  taught  simply  as  methods  and  procedures  to 
be  followed.  Although  guidelines  can  help  commanders  achieve  baseline  levels  of  performance, 
achieving  an  expert  level  of  proficiency  requires: 

•  Extensive  practice  of  command  reasoning  skills  in  a  variety  of  situations  to  acquire  and 
reinforce  skills  until  they  can  be  applied  flexibly  and  automatically.  This  experience  can 
include  a  combination  of  actual  combat,  live  training  exercises,  computer-based  and 
paper-based  simulations,  and  mental  exercises. 

•  Coaching  and  feedback  from  instructors  to  identify  strengths  and  weaknesses  in 
performance,  encourage  reflection,  stimulate  the  student's  thought  processes,  and  enhance 
the  student's  reasoning  capabilities. 

Coaching  and  feedback  can  be  provided  by  expert  instructors  during  dialogs  with 
students.  An  example  of  this  type  of  instruction  includes  Think  Like  a  Commander  (TLAC) 
materials  (U.S.  Army  Research  Institute,  2001).  These  TLAC  materials  are  currently  used  in 
officer  courses  at  Fort  Leavenworth  and  Fort  Knox.  Another  example  is  Tactical  Decision 
Games  (TDGs)  (e.g.,  Schmitt,  1994).  In  both  examples,  tactical  scenarios  are  presented  to 
learners  who  must  then  reach  conclusions  and  make  decisions.  If  the  learner  fails  to  take  an 
applicable  theme  into  account,  the  instructor  asks  the  student  questions,  at  first  indirect,  to 
encourage  the  student  to  reflect  upon  his  or  her  thinking  about  a  theme. 

An  inherent  limitation  of  mentored  instruction  is  that  the  expert  mentors  are  an  expensive 
limited  resource  that  is  infrequently  available;  when  available  their  attention  must  often  be 
shared  among  too  many  learners  for  truly  effective  personalized  interaction.  Classroom 
instruction,  such  as  Army  officer  training,  adds  the  constraint  that  instruction  is  only  accessible 
to  learners  who  can  attend  the  course  on  site,  requiring  significant  time  and  travel  costs  as  well  as 
scheduling  difficulties.  Automated,  computer-based  training  can  provide  instruction  "any  time, 
any  place."  However,  the  types  of  expert  coaching  and  mentoring  required  to  teach  these 
command  reasoning  skills  require  far  more  sophisticated  reasoning  than  is  possible  with  current 
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computer-based  training  (CBT)  and  web-based  training  (WBT)  technologies  that  typically  assess 
learners  based  on  simple  multiple-choice  and  fill-in-the-blank  questions. 

Intelligent  Tutoring  Systems  (ITSs)  are  computer-based  training  systems  that  mimic 
human  instructors  to  provide  automated,  one-on-one  instruction.  The  ITSs  encode  subject  matter 
and  instructional  expertise  to  assess  each  learner's  performance,  knowledge,  and  skills  to  provide 
individualized  learning  experiences,  adapted  to  each  learner's  skills,  background,  and  learning 
preferences.  Most  ITSs  provide  a  simulation  or  problem-solving  user  interface  that  enables 
learners  to  solve  problems  or  pursue  goals  by  entering  a  sequence  of  actions  or  decisions  selected 
from  a  wide  range  of  possible  choices.  The  ITS  applies  subject  matter  expertise  to  evaluate  these 
actions  and  decisions  to  assess  the  learner's  knowledge  and  skills  and  then  select  and  deliver 
appropriate  instructional  interventions. 

However,  no  ITS  system  developed  to  date  supports  a  domain  nearly  as  complex  as 
battlefield  command  which  requires  sophisticated  reasoning  about  a  large  number  of  goals, 
potential  plans,  variables,  and  constraints.  Evaluation  of  each  learner's  solution  is  extremely 
challenging  because  there  is  no  single  "correct"  solution  to  each  scenario.  As  Gen.  George  S. 
Patton  wrote,  “There  is  no  approved  solution  to  any  tactical  situation.”  Thus,  developing  an  ITS 
for  teaching  battlefield  command  reasoning  skills  will  require  "pushing  the  envelope"  in  ITS 
technology  along  a  number  of  dimensions,  such  as: 

•  Representing  solutions  and  their  rationale  to  complex  battlefield  command  problems, 

•  Assessing  the  learner's  reasoning  skills  by  comparing  each  learner's  solution  with 
multiple  expert  solutions  and  their  rationale,  and 

•  Implementing  learner-mentor  interactions  that  enhance  high-level  thinking  habits,  based 
on  an  assessment  of  each  learner's  reasoning,  as  well  as  knowledge  structures  that 
represent  each  expert's  reasoning. 

Opportunity.  We  propose  to  proceed  with  development  of  a  Command  Mentoring 
Intelligent  Tutoring  System  (ComMentor)  based  on  the  observations  and  designs  generated 
during  our  Phase  I  effort.  As  demonstrated  by  our  Phase  I  prototype,  ComMentor  will  present 
TDG  scenarios  to  the  learner,  and  then  manage  a  Socratic  dialog  discussing  the  scenarios  with 
the  learner.  The  system  will  interpret  and  evaluate  the  learner's  solution,  and  then  interact  with 
the  learner  to  stimulate  his  or  her  reasoning  skills  and  refine  its  assessment  of  the  learner's 
strengths  and  weaknesses.  A  combination  of  artificial  intelligence  and  intelligent  tutoring 
system  technologies  provides  enabling  technologies  for  this  system,  such  as: 

•  Scenario-Based  Instruction  -  Stottler  Henke  Associates,  Inc.  (SHAI)  has  pursued 
scenario-based  (or  case-based)  approaches  in  numerous  tutoring  systems  in  which  the 
system's  knowledge  base  contains  a  library  of  problems,  or  scenarios,  to  present  to  the 
learner.  Each  scenario  has  associated  pattern-recognizers  that  evaluate  the  learner's 
decisions  and  actions  and  outcomes  of  the  simulation  to  estimate  the  learner's  mastery  (or 
lack  of  mastery)  of  knowledge  or  skills.  These  mastery  level  estimates  then  provide  the 
basis  for  selecting  instructional  interventions  (e.g.,  lessons,  feedback,  hints,  questions, 
and  stories)  that  prompt  the  learner  to  reflect  on  his  or  her  thinking  and  search  for  and 
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consider  alternatives.  This  scenario-based  approach  differs  from  expert  system/model- 
tracing  approaches  that  generate  problems,  solve  them,  and  then  compare  these 
automatically  generated  solutions  with  those  of  the  learner.  Compared  to  model  tracing 
tutoring  systems,  scenario-based  tutoring  systems  can  teach  skills  in  more  complex 
domains  because  the  system  applies  scenario-specific  knowledge  to  interpret  each 
student’s  actions.  Since  the  range  of  likely  problems  and  plausible  solutions  is  much 
narrower  for  a  single  scenario  than  for  all  possible  scenarios,  scenario-specific  knowledge 
can  be  used  with  relatively  simple  pattern  recognition  algorithms  to  interpret  and  assess 
likely  student  actions  accurately  within  a  given  scenario,  even  in  complex  domains. 

•  Application-Specific  Scenario  Authoring  Tools  -  Any  knowledge-based  software 
system  is  only  as  powerful  and  up-to-date  as  the  set  of  concepts,  rules,  cases,  and  other 
objects  in  its  knowledge  base.  In  large,  complex,  and  continually  evolving  domains,  such 
as  battlefield  command,  practical  development  and  maintenance  of  a  high-performance 
knowledge  base  requires  the  ability  to  capture  most  of  the  knowledge  directly  from 
subject  matter  experts,  without  programming.  Some  tutoring  systems  provide  "generic 
authoring  tools"  that  present  a  standard  user  interface  by  which  the  subject  matter  expert 
must  enter  knowledge  into  the  tutoring  system.  In  many  situations,  however,  subject 
matter  experts  can  enter  scenario  knowledge  much  more  quickly  and  easily  by  using 
knowledge  representations  and  knowledge  editors  that  are  targeted  to  a  specific 
application  domain,  such  as  military  tactical  decision-making.  For  example,  our  Acoustic 
Analysis  Intelligent  Tutoring  System  (AAITS)  under  development  for  the  U.S.  Navy 
captures  the  reasoning  processes  of  undersea  acoustic  analysis  experts  by  enabling  them 
to  annotate  frequency-analyzed  acoustic  data  images  (LOFARGRAMs)  with  features, 
sources,  and  final  classifications  using  a  point-and-click  user  interface.  Thus,  AAITS 
enables  experts  to  maintain  the  tutoring  system's  knowledge  base  of  scenario-based 
problems  and  solutions  by  carrying  out  a  familiar  analysis  activity  in  a  natural  and 
intuitive  way. 

•  Rationale  Capture  -  When  evaluating  each  learner's  solutions,  it's  necessary  to 
understand  the  learner's  reasoning  behind  his  or  her  decisions.  Using  scenario-based 
instruction,  it  is  possible  to  capture  the  reasoning  (or  rationale)  of  experts  for  each 
scenario,  ask  the  learner  questions  that  elicit  his  or  her  reasoning,  and  compare  this 
reasoning  with  those  of  experts.  The  rationale  underlying  expert  solutions  can  also  be 
used  to  stimulate  the  learner's  reflection  and  reasoning  by  identifying  issues  considered 
by  the  expert,  and  then  asking  the  learner  questions  that  lead  the  learner  to  think  about 
those  issues  as  well. 

•  Multi-Modal,  Mixed-Initiative  Conversational  Interaction  -  Our  Phase  I  prototype 
illustrates  the  interaction  of  graphical  and  textual  input  and  output  modalities;  in  Phase  II 
we  intend  to  explore  the  possibility  of  supporting  speech  input  and  output  as  well  (an  area 
of  research  under  current  development  at  SHAI’s  Seattle  office  in  the  Navy  funded 
TALK  project).  With  regard  to  interaction  initiative,  while  it  is  true  that  Socratic  tutoring 
puts  most  control  into  the  hands  of  the  instructor,  that  control  is  highly  conditioned  on  the 
actions  of  the  student;  further,  the  instructor  must  always  be  ready  to  respond  to  a  direct 
query  or  topic  shift  by  the  student  (though  clearly  sometimes  the  right  response  is  to 
dismiss  or  defer  such  student  initiatives).  Again,  SHAI  has  already  begun  to  develop  the 
technology  for  control  of  mixed-initiative  tutorial  dialogs. 
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At  SHAI,  we  have  a  team  uniquely  qualified  to  extend  and  integrate  the  technologies 
identified  above.  During  this  proposed  project,  we  plan  to  draw  upon  our  past  experiences,  and 
those  of  other  researchers  in  these  areas. 

Solution.  The  ComMentor’s  assessment  of  the  student’s  understanding  and  intentions 
will  be  based  on  several  subsystems,  initially  conceived  as  an  analysis  pipeline  that  feeds  its 
outputs  back  to  the  user  interface,  but  actually  implemented  in  Phase  I  as  an  executive  making 
calls  on  two  subordinate  modules  (see  Figure  1  and  Figure  2  for  abstracted  architecture  diagrams 
reflecting  the  Phase  I  design  and  actual  implementation): 

1 .  A  Graphic  Input  Analyzer  (GIA)  that  extracts  situation  descriptions  from  icon 
placements,  area  highlights,  and  path  indications  the  student  draws  on  the  scenario  map; 

2.  A  Text  Input  Analyzer  (TIA)  that  attempts  to  categorize  the  student’s  typed  input  into 
one  of  an  expected  set  of  utterances  (e.g.,  orders,  reports,  requests,  questions,  answers,  or 
assertions).  Those  utterance  expectations  are  modulated  by  the  state  of  the  tutorial  interaction 
(e.g.,  the  question  or  statement  the  tutor  just  made)  and  the  prior  student  assessment. 

3.  A  Domain  Inference  Module  (DIM)  that  applies  rules  from  a  domain  knowledge-base 
to  answer  a  range  of  relatively  concrete  questions,  and  derive  higher  level  situation  descriptions 
from  the  outputs  of  the  GIA  and  TIA. 

4.  A  Case  Assessment  Module  (CAM)  that  uses  the  results  of  input  analysis  and 
inference  to  retrieve  situation-specific  cases  (prototypical  good  and  bad  solutions)  that  serve  as 
hypotheses  about  student  understanding  and  intentions  and  help  direct  the  tutoring  interaction. 

5.  The  Tutor  Interaction  Module  (TIM)  that  uses  the  results  of  all  the  modules  described 
above,  plus  its  own  session  history,  to  choose  a  next  instructor  move,  and  bias  the  future  analysis 
of  the  other  modules. 


Figure  1.  ComMentor  Phase  I  Abstract  System  Architecture  Design. 
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Figure  2.  ComMentor  Phase  I  Abstract  System  Architecture  Implementation. 

The  CAM  is  a  critical  distinguishing  component  that  allows  ComMentor  to  achieve 
appropriate  assessments  of  student  input  while  limiting  the  amount  of  knowledge  engineering 
required.  A  library  of  "good"  and  "bad"  (fragmentary)  solutions  specific  to  a  scenario  is  much 
easier  for  subject  matter  experts  to  provide  than  a  complete  set  of  general  purpose  analysis  and 
assessment  rules.  The  ComMentor  will  use  similarities  between  the  learner's  solution  and  these 
good  and  bad  solutions  to  identify  strengths  and  weaknesses,  respectively,  in  the  learner's 
solution  and,  therefore,  in  the  learner's  underlying  reasoning  skills.  This  initial  evaluation  will 
by  no  means  be  "complete."  Rather,  this  evaluation  will  result  in  an  initial  model  of  the  learner 
skills  employed  within  the  scenario.  This  model  will  be  refined  during  later  interactions  with  the 
learner  such  as  questions  that  probe  into  the  learner's  reasoning. 

The  ongoing  Socratic  interaction  (using  a  combination  of  graphical  and  textual  input  and 
output)  will  refine  the  system's  assessment  of  the  learner's  reasoning  skills  and  stimulate  the 
learner's  thinking.  In  addition  to  the  Phase  I  interaction  style,  in  Phase  II  we  will  also  explore 
other  types  of  interactions  such  as: 

•  War  stories  -  The  tactical  scenario  can  provide  a  launching  point  for  presenting  selected 
stories  or  testimonials  that  illustrate  important,  relevant  lessons.  For  example,  when  the 
tutoring  system  identifies  a  questionable  decision  made  by  the  learner,  the  system  could 
identify  the  underlying  skill  or  thinking  habit  that  relates  to  the  decision  and  then  suggest 
a  relevant  war  story  to  the  learner  that  relates  to  the  skill  or  decision.  The  key  technical 
challenge  is  indexing  these  stories  effectively  so  that  the  system  can  identify  stories  that 
are  relevant  to  the  student's  solution  or  to  the  current  state  of  the  student-mentor  dialog. 

•  Speech  Input/Output  -  Spoken  language  is  generally  an  easier  and  more  natural 
medium  of  information  exchange  for  people  than  is  typed  text.  It  is,  however,  a  more 
difficult  medium  for  computers  to  cope  with.  Nonetheless,  efforts  in  this  direction  are 
under  way,  including  a  project  at  SHAI.  We  note  that  Forbus,  et  al.  (2000)  has  already 
produced  a  prototype  system  that  combines  sketching  and  speech  input  to  capture 
proposed  course  of  action  (CO As).  However,  feedback  from  military  personnel 
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concerned  with  operational  applications  has  led  them  to  de-emphasize  the  speech 
component  of  their  work  (Forbus,  personal  communication,  2001). 

Again,  a  key  characteristic  of  our  approach  is  our  reliance  on  multiple  recognizably  good 
and  bad  scenario  solutions,  or  solution  fragments,  provided  by  experts,  to  support  comparison 
with  and  evaluation  of  the  learner's  solution.  Use  of  multiple  solutions  increases  the  probability 
that  the  system  can  match  a  portion  of  the  learner's  solution  with  a  portion  of  an  expert's.  In 
Phase  II  we  will  devote  substantial  effort  towards  developing  authoring  tools  that  enable  domain 
experts  to  enter  good  and  bad  solutions  (decisions  and  rationale)  intuitively,  using  an  interface 
that  is  similar  to  the  interface  used  by  learners.  The  idea  is  that  experts  could  then  annotate  such 
solutions  by  identifying  skills  that  are/are  not  demonstrated  by  the  learner  if  the  learner's  solution 
matches  part  of  the  expert's  good  or  bad  solution.  This  scenario-specific  analysis  will,  as 
sketched  above,  then  be  combined  with  other  types  of  knowledge,  including  generic  instructional 
strategies,  scenario-specific  instructional  strategies,  and  tactical  "common  sense." 

Benefits  of  our  Approach 

Benefits  of  our  approach  include: 

•  Intelligent  automated  instruction  of  battlefield  command  reasoning  -  As  with  most 
automated  instructional  systems,  our  approach  will  enable  learners  to  learn  "any  time, 
any  where,"  at  less  expense  and  with  more  scheduling  flexibility.  However,  unlike  CBT, 
WBT,  and  simpler  ITSs  developed  to  date,  our  approach  will  support  sophisticated 
learner-mentor  interactions  that  stimulate  the  learner's  thinking  skills  for  an  extremely 
complex  domain:  battlefield  command  reasoning. 

•  Feasible  scenario-based  knowledge  acquisition  -  Our  scenario-based  approach  will 
enable  the  intuitive  entry  of  subject  matter  expertise  without  programming,  using 
rationale  capture  techniques  and  application-specific  scenario  authoring  tools.  By 
making  it  possible  to  create  new  scenarios  quickly,  it  will  be  feasible  to  create  a  rich 
library  of  training  scenarios  that  covers  a  wide  range  of  situations  and  learning  objectives. 
By  making  it  feasible  to  enter  many  solutions  for  a  given  scenario,  the  system  will  be 
able  to  draw  upon  the  perspectives  of  many  different  experts  to  evaluate  each  learner's 
solutions  more  accurately. 

•  Adaptive,  flexible  instruction  -  Our  system  will  employ  both  generic  and  scenario- 
specific  instructional  strategies  and  principle-based  instruction  to  support  instructional 
interactions  that  are  highly  adaptive  to  the  learning  needs  of  each  learner.  By  carefully 
separating  the  system's  scenario-based  tactical  analysis  knowledge  from  the  system's 
instructional  strategies,  it  will  be  possible  for  different  instructional  strategies  to  exploit 
the  same  tactical  knowledge  in  many  different  ways,  to  achieve  diverse  instructional 
goals,  depending  upon  each  learner's  knowledge,  skill  levels,  and  learning  preferences. 

Innovations 

The  ComMentor  will  be  the  first  implementation  of  an  ITS  that  supports  the  teaching  of 
such  complex  and  crucial  high-level  skills.  Bringing  together  the  scenario  and  rationale 
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representation,  the  user  interfaces,  and  the  domain  knowledge  will  stretch  the  limits  of  what  has 
been  done  in  ITS,  and  for  that  matter,  in  artificial  intelligence  generally.  We  believe  that  the 
practical  implementation  of  tutoring  systems  for  large,  complex  domains  will  be  enabled  by 
avoiding  "deep,"  expert  system  representations  of  domain  concepts  in  favor  of  scenario-specific 
(case-based)  knowledge  representations  that  are  "deep  enough."  Specifically,  we  believe 
strategic  combination  of  rule  based  and  case  based  reasoning,  annotation  of  cases  with  rationales, 
and  inclusion  of  multiple  good  and  bad  solutions  and  stories  in  the  case  library,  together  form  a 
cluster  of  techniques  that  will  enable  a  practical  solution  to  a  currently  unsolved  problem. 

Relevant  Methodologies 

Scenario-Based  Instruction 

The  validity  of  the  scenario-based  approach  has  both  intuitive  appeal  and  empirical 
backing.  As  early  as  1940,  Gragg  (1940)  argued  for  scenario-based  instruction.  By  presenting 
scenarios  that  illustrate  the  important  principles,  the  learner  can  see  how  principles  are  applied  in 
operational  contexts  and  tasks.  It  also  defeats  the  well-known  problem  of  inert  knowledge  first 
described  by  Whitehead,  (1929)  and  frequently  validated  by  other  researchers.  Inert  knowledge 
is  information  or  principles  that  a  subject  knows  and  can  recall,  but  which  he  does  not  apply 
when  the  situation  clearly  calls  for  it.  Scenario-based  instruction  (and  related  concepts  such  as 
example-based  instruction,  anchored  instruction,  case-based  instruction,  simulation-based 
instruction,  and  situated  instruction)  overcomes  this  problem  by  showing  learners  the  application 
of  principles  in  an  operational  setting  and  forcing  them  to  apply  them  as  well. 

Socratic  Tutoring  Systems 

Socratic  tutoring  is  a  recognized  style  of  instruction,  often  associated  with  expert  tutors 
(e.g.,  Glass,  1999),  and  known  to  produce  effective  outcomes  (e.g.,  Rose,  Moore,  VanLehn  & 
Albritton,  2000).  Adopting  the  Socratic  style  is  not  a  simplistic  tutoring  “trick” —  it  is  not  about 
syntactically  transforming  all  tutor  interventions  into  question  form.  Rather,  it  is  best  thought  of 
as  a  bias  towards  having  the  student  construct  as  much  of  their  knowledge  as  possible,  while  also 
encouraging  reflection  and  analysis.  While  there  has  been  ongoing  interest  in  automating 
Socratic  tutoring  over  the  last  three  decades,  there  are  currently  only  a  handful  of  leading 
laboratories  with  a  discernable  focus  in  this  direction  (e.g.,  Martha  Evens  at  IIT,  Art  Graesser  at 
Memphis,  Johanna  Moore  at  Edinburgh,  and  sometimes,  Roger  Schank  late  of  Northwestern  and 
now  Cognitive  Arts). 

One  observation  to  be  drawn  from  past  and  current  work  is  that  most  developers  see 
unstructured  language  input  as  a  useful  complement  to  a  Socratic  tutoring  approach.  The  general 
hypothesis  is  that  much  of  the  value  in  Socratic  tutoring  lies  in  having  students  work  out  for 
themselves  what  is  going  on  in  a  given  problem  (thus  leveraging  the  benefits  of  constructivist 
approaches  and  “self  explanation”)  which  would  be  somewhat  short-circuited  by  pre-structuring 
lists  of  possible  answers  (or  answer  components)  as  in  a  menu-driven  input  system.  For  these, 
and  other  reasons  related  to  tutoring  behaviors  observed  in  our  sample  sessions  we  chose  to 
pursue  a  natural  language  interface  in  our  Phase  I  effort. 
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Natural  Language  Input  Processing 

Natural  Language  Processing  (NLP)  has  been  an  active  area  of  artificial  intelligence 
research  since  the  beginning  of  the  discipline.  Much  work  in  the  field  draws  on  well-established 
linguistic  traditions — adopting  a  multi-phased  analysis  mechanism  that  starts  by  emphasizing 
syntactic  parsing.  However,  much  of  the  most  effective  work  in  the  field  has  taken  alternate 
approaches — emphasizing  semantic  analysis  tied  to  some  particular  context  of  language  use. 

This  often  leads  to  faster,  more  useful  processing,  that  is  robust  in  the  face  of  real-world  input 
(ungrammatical,  misspelled,  or  telegraphic  input).  In  fact,  the  dominant  trend  in  Socratic 
tutoring  language  components  is  to  emphasize  more  effective  semantic  analysis  techniques  (e.g.. 
Brown  &  Burton,  1975;  Glass,  1999;  Wiemer-Hastings,  P.,  Wiemer-Hastings,  K.,  &  Graesser, 
1999).  We  note,  however,  that  Glass  (1999)  has  effectively  critiqued  the  use  of  Latent  Semantic 
Indexing  (LSA)  based  approaches  (as  in  AutoTutor)  as  unlikely  to  be  very  effective  in  domains 
that  go  far  beyond  general  discussions  of  terminology  and  definitions,  because  LSA  matching  is 
primarily  sensitive  to  overlaps  in  word  meaning  rather  than  to  structured  relationships  among 
actions  and  situations  so  critical  to  scenario-based  instruction.  Our  language  interpretation 
approach  is  able  to  make  appropriate  use  of  the  structure  typically  available  in  language. 

Case-Based  Situation  Interpretation/Diagnosis 

Our  aim,  in  fact,  is  to  go  far  beyond  instruction  on  terms  and  definitions.  We  aim  to  tutor 
students  in  effective  application  of  high-level  reasoning  skills  in  the  complex  domain  of  dynamic 
battlefield  command.  Tuning  our  language  processing  approach  to  extract  structure  descriptions 
is  not  enough;  all  of  our  student  modeling  and  assessment  must  be  capable  of  making  relatively 
fine  situational  distinctions.  This  is  where  we  believe  the  technology  of  case-based  situation 
interpretation  and  diagnosis  can  be  usefully  applied.  Our  experience  in  building  ITSs  for  similar 
domains  shows  that  constellations  of  fragmentary  cases  can  be  used  to  effectively  diagnose  many 
student  successes  and  failures  in  complex  tactical  scenarios.  Furthermore,  a  case-based  approach 
to  building  such  recognizers  tends  to  make  it  easier  for  domain  experts  to  manage  and  maintain  a 
tutoring  system  over  time. 

Application-Specific  Authoring  Tools 

We  hold,  along  with  most  of  the  ITS  community,  that  a  critical  component  of  real-world 
success  for  ITS  technology  will  be  the  ability  to  provide  tools  that  let  domain  experts  take  more 
control  of  production  and  maintenance  of  training  systems.  We  have  already  noted  that  any 
knowledge-based  software  system  is  only  as  powerful  and  up-to-date  as  the  set  of  concepts, 
rules,  cases,  and  other  objects  in  its  knowledge  base.  Knowledge  in  such  systems  tends  to  have  a 
limited  shelf-life  as  the  world  changes  (e.g.,  new  weapons  and  vehicles,  new  operational 
concepts,  new  doctrine)  and  goals  for  the  system  evolve  (e.g.,  shift  from  focusing  on 
conventional  conflict  to  focusing  on  missions  other  than  war).  To  remain  effective  in  the  long 
run,  a  system  must  be  maintainable  and  extensible;  to  remain  cost-effective,  as  much 
maintenance  and  authoring  as  possible  should  be  supportable  by  available  staff  and  those  with 
first-hand  expertise  in  the  domain. 
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Raw  manipulation  of  the  underlying  data  structures  that  enable  an  ITS  to  function  tend  to 
require  detailed  expertise  in  artificial  intelligence  (AI)  and  ITS  technologies.  To  enable  domain 
experts  to  take  on  maintenance  and  authoring  tasks,  manipulations  of  system  structures  must  be 
translated  into  terms  that  are  meaningful  in  the  domain.  Furthermore  operations  must  be 
clustered,  sequenced,  and  modeled  so  that  appropriate  system  modifications  become  easy,  and 
invalid  modifications  become  essentially  impossible.  Achieving  these  ends  requires  creation  of 
application-specific  authoring  tools.  Since  we  see  this  as  an  important  consideration  in  every 
ITS  we  construct,  this  is  an  area  where  SHAI  has  had  extensive  experience. 

Phase  I  Objectives  and  Accomplishments 


Phase  I  Objectives 

During  Phase  I,  our  objectives  were  to: 

1 .  Identify  the  kinds  of  tactical  analysis  knowledge  that  should  be  represented  by  the 
system  to  support  the  evaluation  of  learner  solutions  to  tactical  scenarios,  in  order  to  refine  the 
system's  student  model. 

2.  Explore  and  evaluate  candidate  interaction  techniques  that  can  achieve  the  types  of 
instructional  goals  currently  achieved  in  leamer/mentor  interactions  with  expert  human 
instructors. 

3.  Identify  the  types  of  automated  tutoring  strategies  needed  to  select  and  generate 
effective  instructional  interventions  based  on  a  student  model. 

4.  Develop  methods  for  authoring  ITS  scenarios  by  entering  good  and  bad  sets  of 
decisions  and  rationale,  and  then  annotating  these  solutions  with  skills  that  are  demonstrated 
(positively  or  negatively)  by  each  solution. 

5.  Develop  a  limited,  proof-of-concept  software  prototype  that  illustrates  key  ideas 
related  to  leamer/mentor  interaction  methods,  tactical  analysis  knowledge  representation,  and 
interactive  tutoring  strategies. 

Phase  I  Activities  and  Accomplishments 

Our  core  activities  during  the  Phase  I  effort  included: 

a.  Study  of  prior  work  related  to  the  project  goals  (e.g.,  reasoning  skills  training,  ITS 
techniques  and  Socratic  Tutoring,  and  the  existing  Think  Like  a  Commander  course). 

b.  Conduct,  observation,  and  analysis  of  a  series  of  Tactical  Decision  Game  tutoring  sessions 
between  active-duty  Marine  Captains  and  acknowledged  tactical  (and  tutoring)  experts. 

c.  Design  and  construction  of  a  proof-of-concept  ComMentor  prototype  based  on  a  single 
TDG  scenario  covered  in  multiple  tutoring  sessions. 


9 


With  respect  to  our  original  objectives,  activities  ‘a’  and  ‘b’  directly  addressed  objectives 
1, 2,  and  3.  Our  early  work  on  activity  ‘a’  helped  frame  the  problem.  The  extensive 
observations  and  analysis  in  activity  ‘b’  ended  up  forming  the  primary  basis  for  most  of  our 
work;  identification  of  knowledge  form  and  content,  interaction  techniques,  and  tutoring 
strategies  all  fed  forward  into  later  objectives.  Activity  ‘c’  directly  addressed  objectives  4  and  5. 
Our  work  on  objective  4  was  not  only  a  necessary  precondition  to  success  on  objective  5,  but 
forms  the  groundwork  for  Phase  II  work  on  task-specific  authoring  tools.  Likewise  the  prototype 
produced  for  objective  5  not  only  demonstrates  the  feasibility  of  our  ideas,  but  also  serves  as  a 
design  base  and  source  of  lessons  learned  for  our  proposed  further  system  development  during 
Phase  II. 

ARI  Direction  and  Materials 

Discussions  during  the  project  kickoff  meeting  held  at  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI’s)  Fort  Knox  office,  in  combination  with  review  of  the 
Lussier,  Ross,  &  Mayes  (2000)  provided  our  initial  direction  on  the  project. 

Much  discussion  focused  on  ARI’s  efforts  to  develop  the  Think  Like  a  Commander 
TLAC  course,  as  an  example  of  what  training  in  battlefield  command  reasoning  skills  might  look 
like.  We  were  given  access  to  initial  materials  on  TLAC,  including  preliminary  instructors’ 
notes  being  compiled  by  Col.  Fontenot  under  contract  to  ARI  at  Fort  Leavenworth.  It  was 
suggested  that  we  might  want  to  base  our  analysis  and  development  efforts  on  the  TLAC 
materials,  and  that  we  might  want  to  follow  a  new  direction  ARI  was  establishing  to  retarget 
TLAC  materials  towards  training  Captains  (rather  than  Colonels). 

Following  up  on  this  initial  meeting,  we  were  provided  with  some  of  TLAC  scenario  and 
vignette  briefing  materials,  as  well  as  videotapes  of  TLAC  class  sessions  run  at  Fort 
Leavenworth.  Ultimately,  we  were  invited  to  join  a  group  of  other  contractors  in  visiting  Fort 
Leavenworth  to  observe  TLAC  courses  for  a  day.  We  observed  two  classes,  each  about  50 
minutes  long.  Interestingly  (disappointingly),  from  the  perspective  of  our  goal  to  develop  a 
tutoring  system,  the  TLAC  class  sessions  involved  between  1 0  and  20  students  at  a  time,  and 
spent  only  about  15-20  minutes  on  each  vignette.  As  a  follow-up  to  this  trip,  we  received  a  more 
extensive  packet  of  material  intended  to  provide  TLAC  students  with  a  detailed  background 
scenario  supporting  interpretation  of  the  several  TLAC  vignettes. 

Finding  the  available  TLAC  observations  (video  and  live)  limited  in  their  usefulness  for 
our  tutoring  analysis  purposes,  we  had  been  planning  to  stage  our  own  simulated  TLAC  sessions 
using  our  domain  expert  consultants  as  instructors.  Our  experts,  however,  were  more 
comfortable  working  with  their  own  TDG  materials,  since  they  found  the  overarching  TLAC 
scenario  (with  its  staggering  wealth  of  documentation)  difficult  to  master  and  not  significantly 
more  conducive  to  enabling  effective  training  in  battlefield  command  reasoning  skills.  We  had 
also  been  studying  TDG  material  (e.g.,  Schmitt,  1994)  as  part  of  our  project  effort.  Fortunately, 
ARI  had  made  it  clear  that  TLAC  was  being  offered  as  an  example  and  a  resource,  rather  than  a 
dictate,  so  we  were  free  to  take  what  we  had  learned  from  our  studies  of  TLAC  and  transfer  it  to 
the  TDG  framework. 
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TDG  Observations  and  Analysis 

Our  experience  with  TDGs  began  with  a  sample  session  during  which  staff  from  SHAI 
and  our  psychology  subcontractor  played  “student”  to  one  of  our  subject  matter  experts  as 
“instructor”;  we  played  the  “Tanks  on  the  Farm”  (ToF)  scenario.  This  was  a  valuable  experience 
for  several  reasons.  First,  it  introduced  us  to  the  on-line  distributed  format  and  infrastructure  that 
our  subject  matter  experts  had  developed  for  electronic  TDG  training;  we  found  this  to  be  a 
convenient,  cost-effective,  and  pedagogically  effective  approach.  Second,  it  introduced  us  to  the 
particular  TDG  scenario  that  would  ultimately  become  the  focus  for  the  bulk  of  our  Phase  I 
analysis  and  implementation.  Third,  it  demonstrated  convincingly  that  we  needed  real  military 
personnel  to  play  the  role  of  student  if  we  wanted  to  observe  more  interesting  tutoring  behavior. 

Based  on  their  expertise  and  reputation  as  outstanding  tactical  instructors,  our  subject 
matter  experts  were  able  to  recruit  active-duty  Marine  Captains  (themselves  all  tactical 
instructors)  to  serve  as  students  in  our  tutoring  observations.  In  all,  we  held  six  observed 
tutoring  sessions,  each  about  an  hour  and  a  half  in  length.  Our  two  experts  took  turns  as  lead 
instructor:  generally  the  lead  instructor  would  handle  the  first  45  minutes  or  so  of  the  session, 
then  the  alternate  instructor  would  weigh  in  for  about  30  minutes  covering  topics  they  thought 
had  been  missed,  and  finally  the  lead  instructor  would  close  out  the  session.  In  all  cases,  one  of 
our  subject  matter  experts  was  physically  on  site  with  the  student,  while  all  other  participants 
were  spread  across  the  country,  observing  manipulations  of  a  scenario  map  on  a  shared 
whiteboard,  and  listening  to  audio  over  a  teleconference  line.  Four  different  students  were 
involved.  Three  different  scenarios  were  covered  in  these  sessions,  offering  a  range  of 
complexity.  We  devoted  three  sessions  to  the  ToF  scenario,  two  sessions  to  a  somewhat  more 
complex  “Defense  of  Glosson”  scenario,  and  one  session  to  the  very  complicated  “Evacuation 
from  Enniottu  City”  scenario. 

For  all  six  of  the  formal  observational  sessions  we  had  two  observers  from  our 
psychology  subcontractor,  and  at  least  one  from  SHAI.  Staff  from  ARI  at  Fort  Knox  joined  us  in 
observing  the  final  session.  Video  and  audio  recordings  were  made  of  the  computer  screen  and 
phone  conversations  in  all  cases.  Rough  on-the-fly  transcripts  were  created  in  real  time. 
Ultimately,  we  focused  our  analysis  on  the  three  ToF  sessions,  compiling  detailed  annotated 
transcripts  of  the  proceedings.  Samples  of  these  are  included  as  Appendix  B  at  the  end  of  this 
report. 

ComMentor  Design  and  Implementation 

As  the  tutoring  sessions  progressed,  we  began  to  refine  our  ideas  on  the  design  of  the 
Phase  I  ComMentor  prototype.  We  identified  a  finer  set  of  distinctions  in  the  knowledge 
required  to  support  robust  tutoring.  We  identified  a  modular  breakdown  appropriate  to  these 
interacting  forms  of  knowledge.  We  began  the  process  of  implementing  the  identified 
knowledge  bases  and  processing  modules. 
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Phase  I  Findings 


The  findings  from  our  Phase  I  effort  can  be  categorized  under  three  divisions,  mirroring 
the  activities  just  described:  (a)  Study  of  ARI  directions  and  Materials;  (b)  TDG  observations 
and  analysis;  and  (c)  ComMentor  design  and  implementation. 

Findings  Based  on  ARI  Direction  and  Materials 

We  took  away  a  small,  but  important  set  of  lessons  from  our  early  survey  of  the  problem 
and  prior  work: 

1.  There  are  vanishingly  few  opportunities  for  officers — especially  junior  grade  officers 
such  as  Captains — to  get  one-on-one  instruction  in  battlefield  command  reasoning  skills.  TLAC 
is  a  very  innovative  course,  but  it  is  available  on  only  a  quite  limited  basis.  At  the  time  we  were 
doing  our  study,  it  was  only  available  to  quite  senior  officers,  only  in  large  group  settings,  and 
only  for  small  amounts  of  time. 

2.  It  is  unclear  that  the  depth  of  background  provided  by  the  overarching  TLAC  scenario 
that  frames  the  smaller  TLAC  vignettes  is  strictly  necessary  if  the  goal  is  to  provide  sufficient 
fodder  for  interesting  battlefield  reasoning  exercises.  The  TDGs  are  exceptionally  concise  in 
presentation,  and  needed  contextual  information  is  either  plausibly  assumed,  or  introduced  by  the 
instructors  on  an  as-needed  basis. 

3.  Prior  work  in  the  context  of  TLAC  had  identified  an  interesting  set  of  analytic  levels 
(or  scaffolding  levels)  at  which  battlefield  command  reasoning  skills  could  be  drilled:  (1) 
Situation  Analysis — i.e.,  “What  do  you  know?”  “What  do  you  think  you  know?”  and  “What  do 
you  need  to  know?”  (2)  Situation  Response — i.e..  Role-playing  the  issuing  of  orders,  reports, 
and  requests  with  feedback  of  relatively  immediate  replies  and  consequences;  and  (3)  Context 
Analysis — i.e.,  analyzing  the  longer-term  and  second-order  effects  of  decisions  made  to  cope 
with  an  immediate  situation,  including  formation  of  future  policies  to  avoid  recurrence  of 
problematic  situations,  or  to  ensure  adequate  flexibility  to  deal  with  them  when  they  arise.  Our 
own  observations  suggested  adding  two  more  levels  of  interaction:  at  the  low  end  (0)  Fact 
Filling — i.e.,  having  the  instructor  fill  in  missing  factual  information  (e.g.,  on  weapons 
capabilities)  and  on  the  high  end  (4)  Behavior  Modeling — that  is,  having  the  instructor  model 
effective  reasoning  and  behavior  after  the  students  have  made  their  attempts. 

4.  When  attempting  to  model  and  simulate  the  actions  of  expert  tutors  in  mentoring 
students  at  a  particular  level  of  expertise,  there  is  no  substitute  for  observing  real  tutoring 
sessions  between  real  experts  and  real  students.  Short  group  sessions  will  not  provide  the 
material  for  modeling  extended  one-on-one  interactions.  Simulations  with  fake  students  will  not 
elicit  the  same  kinds  of  tutoring  behavior  as  will  interactions  with  appropriate  students. 

Findings  Based  on  TDG  Observations  and  Analysis 

The  TDG  tutoring  session  observations  were  central  to  the  entire  direction  and  outcome 
of  our  Phase  I  effort.  Beyond  the  project’s  basic  purpose,  and  the  projected  system’s  form,  and 
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core  underlying  technologies,  essentially  everything  about  the  Phase  I  prototype  was  determined 
based  on  observations  of  these  sessions.  Here,  as  a  main  result,  we  focus  on  our  preliminary 
taxonomy  of  tutoring  moves  as  shown  in  Table  1. 

Table  1  is  broken  into  four  main  blocks: 

1.  Basic  Scenario-Based  Socratic  Tutoring:  This  block  organizes  the  core  set  of 
moves  the  instructor  typically  makes  when  guiding  a  student  through  thinking  about  a  particular 
scenario. 

We  start  by  dividing  essential  factual  information  into  that  which  is  peculiar  to  the 
scenario  at  hand,  and  that  which  is  general  to  the  domain.  Key  instructor  moves  related  to  such 
facts  include  four  active  moves:  (a)  presenting  the  relevant  facts  to  the  student  (e.g.,  introducing 
and  setting  up  the  scenario),  (b)  correcting  student  misstatement  of  facts  (either  directly  or  by 
querying  them),  (c)  clarifying  slightly  misstated  or  incomplete  facts  (again  either  by  statement  or 
query),  and  (d)  recapping  a  set  of  facts.  This  last  variation  on  fact  presentation  is  especially 
interesting,  because  it  can  embed  a  focusing  and  framing  intent — that  is  by  choosing  which  facts 
to  include  or  drop  from  the  recap,  the  instructor  gets  to  focus  the  student  on  essential 
constellations  of  facts.  It  is  characteristic  of  Socratic  tutors  to  often  adopt  a  more  passive  role 
with  respect  to  fact  statement — that  is  to  ask  questions  instead  of  make  assertions.  We  thus 
consider  the  following  pair  of  moves  critical  in  our  analysis:  (e)  probing  for  the  student  to  state  a 
fact,  and  (f)  refining  the  probe  to  get  the  student  to  clarify  an  incorrectly  or  incompletely  stated 
fact. 
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Table  1. 


Phase  I  Analysis  -  Representative  Taxonomy  of  Tutoring  Moves 

1.  Basic  Scenario-Based  Socratic  Tutoring 

Present/correct/clarify/recap  scenario  facts 

Present/correct/clarify/recap  domain  facts  (respond  to  student  questions) 

Probe/refine  for  domain  fact  understanding 
Probe/refine  for  scenario  fact  understanding 
Probe/refine/lead/clarify/recap  for  scenario  situation  assessment 
What  is  key  missing  information? 

What  are  possible  alternatives? 

Probe/refine/lead/clarify/recap  for  decision  details 
Probe/refine/lead/clarify/recap  for  decision  rationale 
Direct  student  action 

2.  Topic  Management 

Instructor-driven 

Explicit  vs.  implicit 
Shifting  vs.  digressing  vs.  abandoning 
Student-driven 

Accepting  vs.  deferring  vs.  rejecting 

3.  Theme-Specific  Tutoring  (examples) 

Focus  on  Mission  &  Higher’s  Intent 

Does  higher  know  what  you  know?  What  should  they  know? 

Has  your  mission  changed?  If  so,  why  and  how? 

Thinking  Enemy 

What  are  they  doing  there?  What  is  their  goal? 

What  might  their  plans  be? 

Big  Picture 

What  don’t  you  know  in  your  situational  awareness  (SA)? 

Visualization 

What  will  happen  after  this  engagement? 

How  will  you  shape  the  future  battlespace? 

Contingencies  and  Flexibility 

Hypothetical  variants  on  base  scenario 

4.  Alternate  Sub-Session  Formats 

Role-play 

Play  at  self  vs.  commander  vs.  enemy,  etc.  (varying  levels  of  competence/personality) 
Behavioral  modeling 
Theoretical  discussion 
Anecdotes 
Motivation 
Reflection 

Ask  student  to  assess  own  performance  (best,  worst) 

_ Ask  student  to  recap  key  learning  points  (what  would  do  different?) _ 
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Moving  beyond  the  facts,  a  tutor/student  interaction  starts  to  focus  on  situation 
assessments  (inferences),  decisions  (intentions),  and  decision  rationales  (reasoning  processes, 
and  decision  justifications).  Some,  but  not  all,  of  the  moves  just  identified  for  facts  apply  to 
these  higher-order  constructs.  In  particular,  tutors,  for  the  most  part,  do  not  present  or  correct 
assessments,  decisions,  and  rationales.  They  are  frequently  observed  probing  for  student 
statements  of  these  constructs  and  refining  their  probes  to  get  at  details  (e.g.,  “Why  did  you  do 
that?’’’’  or  “  What  were  you  thinking ?"  vs.  “Why  are  you  worried  about  that  [protecting your  left 
flank /?”).  The  goal  is  to  get  the  student  to  think,  and  articulate  their  thinking.  One  novel  move 
that  appears  in  this  context  is  the  (g)  leading  question1 — a  tutoring  move  that  stops  short  of 
being  a  presentation  or  correction  (at  least  syntactically),  but  more  or  less  strongly  influences  the 
student  to  focus  on  particular  issues  or  facts,  or  to  see  things  a  particular  way.  Once  the  student 
makes  some  higher-level  statements,  tutors  may  feel  free  to  make  clarifying  or  recapping 
statements  as  they  would  with  facts.  Again,  recapping — as  a  kind  of  editing — can  have  a 
powerful  influence  on  the  future  direction  of  the  session. 

We  have  included  two  sample  generic  questions  in  this  first  block  illustrating  approaches 
to  scenario  situation  assessment.  One  interesting  point  is  that  these  questions  are  very  generic — 
not  tied  at  all  to  any  particular  scenario,  or  any  particular  aspect  of  a  scenario.  Another  point  is 
that  these  questions  could  equally  well  appear  elsewhere  in  the  table— in  particular  as  examples 
of  pursuing  particular  battlefield  reasoning  themes  sampled  in  the  third  block  of  Table  1. 
Essentially  the  same  instructor  behavior  can  be  motivated  in  different  ways,  and  be  intended  to 
serve  different  ends,  depending  on  circumstances. 

The  final  kind  of  tutoring  move  we  include  in  this  section  is  the  explicit  directive  to 
manage  student  action.  Typical  examples  would  be  “ Issue  your  orders''’  or  “ Draw  me  a  picture 
of  that." 


2.  Topic  Management:  The  second  block  of  Table  1  simply  acknowledges  that  an 
extended  conversation,  such  as  a  Socratic  tutoring  dialog,  has  an  internal  topic  structure,  and 
there  are  often  (overt  or  covert)  moves  to  control  the  topic.  We  start  by  noting  that  a  dialog, 
inherently  offers  the  possibility  for  mixed  control  over  issues  such  as  topic;  however,  we 
continue  to  look  at  things  from  the  instructor’s  perspective. 

Accordingly,  we  suggest  that  the  instructor  can  make  the  following  moves  based  on  their 
own  assessment  of  conversational  flow:  (a)  shifting  the  topic  in  an  orderly  planned  way  (either 
explicitly,  e.g.,  “OK.  Let’s  take  a  different  approach.  Let’s  take  a  look  at  your  tactical  plan."  or 
implicitly,  e.g.,  “ What  do  you  think  about  the  way  you ’ve  got  your  forces  arrayed  to  take  care  of 
his  push  from  the  East?"),  (b)  digressing  to  temporarily  address  a  sub  or  side-topic  while 


1  We  note  that  leading  questions  may  be  used  for  purposes  other  than  getting  the  student  to  see  a  point  the  instructor 
wants  them  to  see  (or  at  least  can  end  up  having  other  effects).  They  can  also  be  used  for  diagnosis,  or  for  setting 
traps  for  the  student  to  fall  into — that  is,  for  verifying  tutor  hypotheses  that  it  is  worth  spending  time  drilling  in  a 
particular  area.  For  example:  11  So  you  think  the  idea  of  the  Assembly  Area  in  Viettiville  is  still  viable  at  this  point?" 
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planning  to  resume  the  current  topic  later,  and  (c)  abandoning  a  line  of  discussion  to  deal  with 
some  other,  presumably  more  important  topic  opportunity  that  arose  in  mid-discussion.2 
We  also  suggest  that  an  instructor  can  respond  to  student  attempts  to  change  the  topic  by  either 
(d)  accepting  the  new  topic,  (e)  deferring  the  new  topic  for  later  discussion,  or  (f)  rejecting  the 
new  topic  outright.  We  actually  have  very  few  examples  of  students  trying  seriously  to  redirect 
their  tutoring  sessions — a  likely  mark  of  their  acceptance  of  the  instructors’  expertise  and 
authority,  and  of  their  willingness  to  be  guided.  It  is  an  open  question  as  to  whether  that  bias 
will  carry  over  to  the  automated  tutoring  context. 

3.  Theme-Specific  Tutoring:  The  third  block  of  Table  1  shows  how  a  set  of  relatively 
generic  (for  the  domain)  questions  can  be  identified  for  some  of  the  specific  themes  central  to 
battlefield  command  reasoning  (e.g.,  a  sampling  of  the  eight  themes  identified  as  underlying 
TLAC).  In  any  given  circumstance,  most  of  the  questions  identified  here  could  be  classified  as 
to  their  purpose  according  to  the  scheme  laid  out  in  Block  1  above  (Basic  Scenario-Based 
Socratic  Tutoring).  It  is,  of  course,  possible  to  get  arbitrarily  more  scenario-specific  in  such 
questions.  The  point,  however,  is  that  it  is  often  possible  to  identify  specific  tutor  moves  with 
specific  curricular  goals  (in  this  case,  eliciting  and  reinforcing  certain  critical  identified  aspects 
of  battlefield  reasoning  thinking  skills),  rather  than  with  abstract  tutoring  strategies. 

4.  Alternate  Sub-Session  Formats:  The  final  block  of  Table  1  highlights  that  even  in 
the  context  of  a  primarily  Socratic  tutoring  session,  there  is  room  to  use  a  wide  variety  of 
pedagogical  techniques.  We  observed  our  instructors  using  all  the  approaches  listed  there.  Most 
obviously,  role-play  was  a  significant  and  adaptable  tool  in  their  kit.  Students  were  variously 
asked  to  play  themselves  (that  is,  commander  of  the  tank  battalion  at  the  center  of  the  ToF 
scenario),  their  superior  commander,  or  various  enemy  commanders.  An  interesting  technique 
used  to  get  students  to  consider  alternatives  and  contingencies,  and  to  model  a  thinking  enemy, 
was  the  suggestion  of  specific  ranges  of  competence  or  personalities  for  enemy  commanders. 
Often  the  play  was  aimed  at  specific  issues  or  answering  particular  questions.  Sometimes,  the 
play  was  extended  to  consider  possible  temporal  sequences,  and  thus  grew  into  small-scale  low- 
fidelity  micro  simulations.  In  such  cases,  preliminary  inspection  suggests  it  is  plausible  to  base 
such  plays  on  local  scenario-specific  scripts. 

As  noted  in  “Findings  Based  on  ARI  Direction  and  Materials,”  we  observed  our 
instructors  sometimes  engaging  in  behavior  modeling — that  is  showing  the  student  how  the 
expert  would  do  something — generally  after  giving  the  student  a  chance,  or  several  chances,  to 
do  the  job  themselves.  Likewise,  we  saw  them  devote  limited  time  to  theoretical  discussions  of 
such  important  concepts  as  shaping  the  battle.  On  occasion,  they  used  personal  anecdotes  to 
make  points  (e.g..  I’ve  been  in  tanks  under  heavy  artillery  fire,  and  it’s  amazing  how  tough  even 
a  little  bit  of  armor  is.). 


2  It  may  be  questionable  how  frequently  it  is  useful  or  possible  to  distinguish  between  digressing  and  abandoning — 
that  is  how  often  an  instructor  really  plans  to  abort  a  line  of  discussion,  versus  how  often  they  simply  get  distracted 
and  don’t  happen  to  remember  to  come  back  to  the  topic.  Discussion  with  one  of  our  subject  matter  experts  suggests 
that  often  what  appears  as  the  instructor  abandoning  a  topic  is  really  a  response  to  subtle  cues  (e.g.,  body  language) 
indicating  that  the  student  is  not  benefiting  from  the  current  line  of  attack.  That  could  be  either  because  the  student 
already  gets  the  point  (so  the  current  tutoring  goal  is  accomplished,  and  it  is  time  to  shift  focus),  or  because  the 
student  is  not  following  the  line  of  discussion  (and  so  it  is  time  to  shift  strategies,  while  still  addressing  the  same 
tutoring  goal). 
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One  point,  that  has  only  been  suggested  so  far,  is  that  there  is  a  definite  large-scale 
structure  to  an  entire  tutoring  session.  Clearly,  it  starts  with  a  presentation  of  the  scenario.  The 
session  generally  proceeds  through  an  elicitation  of  the  student’s  initial  response  to  the  initial 
scenario  challenge.  There  is  a  range  of  possible  responses  that  trigger  focus  on  a  range  of 
possible  discussion  topics.  For  any  given  scenario,  some  topics  tend  to  come  up  earlier  in  the 
discussion,  and  some  topics  tend  to  show  up  late,  if  at  all.  Eventually  the  session  ends,  and  in  all 
cases  we  observed,  the  instructors  closed  by  encouraging  a  period  of  reflection.  That  is,  just  as 
they  had  a  reasonably  tight  script  to  launch  the  session,  they  had  a  script  to  end  it  as  well,  by 
asking  questions  such  as  “How  do  you  think  you  did?”  “  What  do  you  think  you  did  best?”  “ What 
do  you  think  you  did  worst?”  “ What  did  you  learn?”  “What  would  you  do  different  next  time?” 

The  above  is  clearly  a  preliminary  analysis,  but  reflects  the  breadth  of  our  Phase  I 
observations.  The  value  of  this  kind  of  taxonomy,  and  of  accompanying  observations,  such  as  on 
the  gross  structure  of  tutoring  sessions,  is  to  begin  to  provide  a  way  to  think  about  structuring  an 
automated  tutor’s  behavior. 

Findings  From  ComMentor  Design  and  Implementation 

As  we  moved  from  observation  and  analysis  into  design,  we  identified  several  other 
aspects  of  the  tutoring  sessions  that  drove  us  to  evolve  beyond  the  implementation  ideas 
presented  in  our  Phase  I  proposal: 

•  We  observed  that  combining  text  and  graphics  was  very  important  to  duplicating  the 
kinds  of  interactions  we  were  observing:  sometimes  a  map  manipulation  can  clarify  an 
otherwise  ambiguous  utterance,  and  sometimes  a  description  of  rationale  can  clarify  an 
otherwise  mysterious  set  of  moves  on  the  map.  Accordingly,  we  are  devoting  substantial 
resources  to  supporting  multi-modal  interaction  in  the  Phase  I  ComMentor  prototype. 

•  We  observed  the  use  of  simple  “role-play”  snippets  (e.g.,  issuing  of  orders,  consideration 
of  hypothetical  follow-on  scenarios)  and  felt  that  separating  the  tutoring  interface  from  a 
“role-play”  interface  would  help  to  clarify  (both  for  the  student  and  the  system)  what  kind 
of  interaction  was  going  on  at  any  given  moment. 

•  We  observed  the  importance  of  having  a  basic  competence  to  reason  about  simple 
quantity,  time,  distance,  and  force  issues  (what  was  often  referred  to  as  “battlefield 
calculus”);  it  appears  that  while  Captains  can  benefit  tremendously  from  tutoring  in 
battlefield  reasoning  skills,  they  often  need  support  in  managing  the  basic  facts  of  the 
situation. 

These  and  other  points  are  illustrated  by  the  Phase  I  software  prototype. 

Phase  I  Prototype  System 

As  part  of  this  Phase  I  effort,  we  built  a  limited  proof-of-concept  prototype  Socratic  ITS 
for  battlefield  command  reasoning  skills.  We  call  this  prototype  “ComMentor” — the  Command 
Mentoring  system.  This  section  provides  a  detailed  discussion  of  the  goals,  structure,  internals, 
and  behavior  of  the  ComMentor  system. 
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Prototype  Goals 

Our  goals  for  the  Phase  I  prototype  ComMentor  system  included  the  following: 

1 .  Demonstrate  the  possibility  of  automated  Socratic  tutoring  in  a  battlefield  reasoning 
application; 

2.  Demonstrate  the  applicability  of  the  Socratic  tutoring  principles  identified  above; 

3.  Explore  architectural  options  for  an  automated  Socratic  tutoring  system 
implementation; 

4.  Help  develop  requirements  and  designs  for  a  more  complete  Phase  II  prototype. 

General  Discussion  of  Prototype  Scenario 

Our  Phase  I  prototype  presents  a  clear  image  of  the  kind  of  system  we  propose  to 
construct  during  this  Phase  II  effort.  A  lesson  is  centered  on  a  Tactical  Decision  Game  (TDG) 
scenario  presented  by  a  combination  of  a  situation  map,  a  set  of  force  structures,  and  a  textual 
description.  The  description  will  generally  contain  discussion  of  the  Blue  unit  that  the  learner 
commands,  the  learner’s  orders,  intelligence  about  the  enemy,  and  often  some  critical  incident 
that  starts  the  action  in  motion  (a  “trigger  event”  that  triggers  the  need  for  a  decision).  For 
example,  as  described  earlier,  in  Phase  I  we  developed  a  scenario  around  ToF  TDG.  Figure  3  is 
a  version  of  the  scenario’s  introductory  text.  Figure  4  shows  the  ComMentor  screen  layout  with 
the  ToF  situation  map  and  force  structures  displayed. 

At  this  point,  the  student  can  be  prompted  for  a  decision.  Based  on  extensive  experience 
playing  this  game  with  a  wide  range  of  students,  our  domain  experts  indicate  that,  at  a  high  level, 
students  essentially  always  take  one  of  three  approaches: 

1 .  The  student  decides  to  try  evading  the  enemy  force  so  as  to  continue  on  to  the 
assigned  position  at  Viettiville. 

2.  The  student  decides  to  react  to  the  presence  of  the  enemy  force  in  a  cautious  way  that 
involves  significant  preparation  and  maneuvering. 

3.  The  student  decides  to  take  on  the  enemy  force  with  a  rapid  race  to  the  top  of  Balzer 
hill  that  preserves  the  element  of  surprise  for  Blue. 

Any  of  these  responses  provides  an  opening  for  interesting  discussion  and  analysis. 
Typical  issues  that  can  be  well  addressed  in  the  context  of  this  scenario  include: 

•  Interpretation  of  commander’s  intent. 

•  Modeling  a  thinking  enemy. 

•  Assessment  of  the  big  picture. 

•  Visualization  of  an  evolving  battlefield. 
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•  Recognizing  contingencies  and  maintaining  flexibility. 

•  Reasoning  about  time. 

•  Exploitation  of  terrain. 

•  Basic  “battlefield  calculus”  (time,  distance,  and  force  calculations). 


You  command  a  tank  battalion  of  four  companies  of  Ml  A  Is,  with  a  section  of  eight  tube- 
launched  optically-tracked  wire-guided  (TOWs)  mounted  on  high  mobility  multipurpose 
wheeled  vehicle  (HMMWVs).  Your  division  has  been  fighting  elements  of  the  enemy  6th 
Division,  a  battle-hardened  outfit  of  three  maneuver  brigades  supported  by  an  armored 
reconnaissance  battalion  and  an  artillery  regiment.  Typically,  a  reconnaissance  company  and 
artillery  battalion  will  support  each  maneuver  brigade.  Intelligence  estimates  that  10  Brigade, 
the  toughest  formation,  is  at  about  80%  strength  and  that  20  Brigade  is  at  about  50-75%.  After 
having  been  pulled  out  of  action  some  two  weeks  ago  for  refitting,  your  battalion  is  now  once 
more  near  full  strength.  You  are  making  a  night  administrative  move  on  improved  roads  to  your 
new  assembly  area  in  Viettiville,  where  you  will  be  part  of  the  division  reserve.  The  front  is 
about  10-12  km  east  of  the  east  edge  of  the  map.  During  the  move  you  are  instructed  to  observe 
radio  listening  silence. 

The  enemy  has  been  attacking  from  the  east  with  armor  and  mechanized  infantry,  usually 
leading  with  a  battalion  of  tanks  preceded  by  a  reconnaissance  company  on  one  or  more  axes 
depending  on  the  terrain.  Cross-country  mobility  for  armor  is  good,  except  in  wooded  areas, 
which  are  essentially  impassable  to  vehicles.  The  division  commander  feels  the  time  is  about 
right  for  a  counterstrike.  Division  intends  to  use  your  battalion  to  spearhead  a  counterattack  out 
of  Viettiville,  direction  and  objective  to  be  determined  as  the  situation  develops,  in  order  to 
destroy  the  enemy’s  armor,  seize  the  initiative  and  transition  to  the  offensive.  You  are  expected 
to  be  in  Viettiville  by  0630,  about  90  minutes  from  now. 

At  the  head  of  the  battalion  march  column,  you  approach  Balzerton  from  the  southwest  as 
the  day  begins  to  dawn.  It  has  been  an  uneventful  road  march.  During  the  night  you  heard  some 
traffic  on  the  division  command  net  about  skirmishes  with  enemy  infantry,  but  no  major  activity. 
But  then  you  are  contacted  on  the  radio  by  a  section  of  light  armored  vehicle  (LAV)-25s 
performing  rear-area  security.  They  had  passed  you  earlier  on  the  road  and  are  now  in  Balzerton. 
The  light  armor  section  leader  makes  the  following  report: 

I  count  about  25-30  T-72s — I  say  again,  T-72s — at  Vietti ’s  Farm.  They  are  apparently 
laagering;  I  see  dismounted  crews.  I  see  a  pair  ofT-72s,  apparently  scouts,  approaching  Balzer 
Hill  from  the  east;  they  ’ll  be  in  position  to  see  you  in  less  than  five  mikes.  I  spot  a  convoy  of 
trucks — about  a  half  dozen,  including  refuelers — approaching  the  farm  on  the  Viettiville  road, 
about  three  clicks  from  the  farm.  Iam  unobserved. 

The  section  leader’s  voice  is  urgent,  but  not  panicked.  He  is  sure  of  what  he  sees. 


Figure  3.  Tanks  on  the  Farm  Introduction. 
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Welcome  to  the  OomMentor 

\  You  command  a  tank  battalion  of  4  companies  of  Ml  Als.yrtth  a  section 
of  8  TOWs  mounted  on  HMMWVs.  You  have  been  fighting  forces  of  the 
\  enemy  6th  Division,  abattle-harderied  outfit  of  3  maneuver  brigades 
supported  by  an  armored  reconnaissance  battalion  and  an  artilien/ 


Figure  4.  ComMentor  Screen  Layout  w.  ToF  Map  and  Force  Structures. 


Of  course  some  responses  make  it  easier  or  more  natural  to  talk  about  some  of  these 
topics  than  others.  For  instance,  if  the  student  chooses  the  first  option — avoiding  the  pending 
confrontation  in  favor  of  following  the  original  orders — it  is  natural  to  begin  a  discussion  about 
the  big  picture  and  acting  in  concert  with  the  commander’s  intent.  An  important  aspect  of 
understanding  this  scenario  is  to  see  that  the  laagering  tanks  are  likely  to  be  the  lead  battalion  of 
an  enemy  breakthrough  from  the  East.  Given  the  lack  of  radio  traffic  reporting  such  a  major 
incursion  into  Blue  territory,  Division  command  is  likely  not  yet  aware  of  what  is  going  on. 
Furthermore,  enemy  logistics  trucks  coming  down  the  road  from  Viettiville  suggest  the  intended 
assembly  area  is  already  compromised.  In  light  of  all  this,  assembling  at  Viettiville  in  order  to 
launch  a  counterstrike  and  destroy  enemy  armor  is  now  beside  the  point.  The  student  has  an 
opportunity  to  seize  the  initiative  and  start  destroying  enemy  armor  right  here  and  now. 


If  the  student  seems  to  have  chosen  the  second  or  third  option,  the  interaction  goes 
somewhat  differently.  It  is  still  worth  exploring  the  issues  of  situation  assessment  and 
commander’s  intent,  but  the  conversation  now  is  oriented  towards  probing  the  student’s  rationale 
in  order  to  confirm  that  they  took  their  actions  for  the  right  reason.  If  such  probing  confirms  an 
appropriate  rationale,  then  the  discussion  can  move  on  to  other  topics.  For  instance,  if  the 
student  has  taken  a  slow  cautious  approach,  the  discussion  can  turn  to  appropriate  use  of  time 
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and  exploiting  the  element  of  surprise;  among  other  possible  directions,  this  may  lead  to 
reviewing  issues  of  “battlefield  calculus”  (e.g.,  time,  distance,  and  force  calculations).  If  the 
student  has  moved  quickly  and  decisively  to  destroy  the  laagering  tanks,  then  after  another 
rationale  confirmation  discussion,  the  tutor  might  focus  on  the  specifics  of  their  plan:  e.g.,  the 
deployment  of  available  forces,  effective  use  of  terrain,  awareness  of  possible  ambushes,  and 
even  the  form  in  which  orders  were  issued. 

Given  enough  guidance,  pretty  much  any  student  can  be  led  to  an  understanding  of  the 
big  picture,  and  to  an  appreciation  of  the  value  and  appropriateness  of  decisive  action.  But  there 
is  potentially  still  more  to  this  scenario.  An  assessment  of  the  big  picture  raises  questions  about 
the  larger  enemy  element  the  laagering  T-72s  belong  to,  and  that  larger  enemy’s  likely 
intentions.  It  raises  questions  about  the  fight  after  the  current  fight,  and  the  relation  of  the 
current  situation,  not  only  to  the  commander’s  original  intent,  but  also  to  the  commander’s 
situation  awareness  and  likely  future  intent.  This  relatively  simple  scenario,  thus  serves  as  a 
context  in  which  to  address  a  large  number  of  battlefield  reasoning  themes. 

Points  Illustrated  by  Phase  I  Prototype 

The  Phase  I  prototype  as  demonstrated  at  ARI  on  7  June  2001  illustrates  many  of  the 
points  central  to  our  approach  to  building  Socratic  ITSs  for  high-level  reasoning  skills.  In  this 
section  we  will  review  those  points  and  the  prototype  capabilities  that  relate  to  them. 

Our  observations  of  tutoring  sessions  convinced  us  of  the  importance  of  supporting  multi¬ 
modal  input  and  output.  In  those  sessions,  the  primary  modes  were  graphics  (task  organizations 
and  manipulable  maps)  and  speech.  In  our  implementation  we  aimed  to  duplicate  the  graphics 
capabilities  and  substitute  natural  language  text  for  the  observed  speech3.  The  scenario 
introductory  sequence  illustrates  the  ability  of  the  system  to  output  both  text  and  graphics.  The 
text  is  currently  output  as  one  large  block  into  the  “Tutor”  pane  (refer  again  to  Figure  4).  The 
system  is  capable  of  directing  individual  blocks  of  text  to  any  of  its  text  output  panes  (e.g.,  the 
role-play  channels  for  “Battalion,”  “Division,”  or  the  “LAVs”).  On  the  graphic  side,  the  task 
organizations  for  both  Red  and  Blue  are  displayed  as  (expandable/contractable)  hierarchical  trees 
with  appropriate  icons  (the  upper  right-hand  column  in  Figure  4).  The  underlying  situation  map 
is  a  prepared  image  file,  but  the  placement  of  moveable  items  such  as  the  Red  and  Blue  units  are 
under  program  control  (in  the  upper  left-hand  block  of  Figure  4). 

Multi-modal  input  (graphics  &  text) 

The  prototype  illustrates  an  example  of  the  multi-modal  input  capabilities  we  identified 
as  important  to  a  naturalistic  tutor.  On  the  text  side,  we  showed  that  the  system  can  parse  orders 
to  units  and  instantiate  underlying  assignment  structures.  On  the  graphics  side,  we  showed  that 
the  student  could  manipulate  the  icons  on  the  map,  dragging  them  to  new  locations  as  desired  to 
help  illustrate  a  plan.  The  system  is  able  to  compute  a  range  of  simple  spatial  predicates  to 
describe  icon  placement,  and  uses  those  to  attempt  to  verify  that  the  icon  locations  match  up  with 


3  As  noted  during  our  presentation,  it  would  not  have  been  tremendously  hard  to  demonstrate  speech  output  in  our 
prototype,  but  resource  limitations  argued  against  it.  The  proposal  for  Phase  II  addresses  the  issue  of  speech  input. 
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the  orders  previously  given  in  English.  Thus  the  system  can  use  information  acquired  from  these 
two  input  modalities  together,  to  build  a  more  robust  understanding  of  the  student’s  intentions. 

Free-form  input  (natural  language) 

When  it  came  to  inputting  orders,  we  demonstrated  the  ability  to  parse  rather  varied  and 
complicated  expressions  for  the  assignments  that  represent  the  student’s  plan  for  his  units.  Our 
domain  experts  consider  language  a  far  superior  form  for  initial  input  as  compared,  say  to  menu 
systems  or  map  overlay  manipulation,  because  it  corresponds  closely  to  the  real  task,  and  neither 
leads  the  student  to  a  likely  answer,  nor  promotes  inappropriate  abstraction  of  moving  game 
tokens  on  a  board.  In  our  demonstration  sequence  we  input  a  set  of  orders  roughly  as  follows: 

Alpha  go  to  Balzerton  and  observe  laagering  tanks.  Bravo  stay  west  of  Balzer  Hill  and 
wait  in  reserve.  Our  tows  should  also  stay  west  of  Balzer  and  wait.  Charlie  move  onto 
Balzer  Hill  and  fire  on  enemy  scouts  when  in  range.  Delta  move  south  of  the  hill  and 
observe  laager  formation. 

While  our  parser  is  not  yet  a  robust  capability,  it  illustrates  the  promise  of  taking  a 
situated  semantics-first  approach  to  input  text  interpretation.  Similarly  we  were  able  to  parse  a 
range  of  factual  and  hypothetical  statements  about  the  situation,  as  well  as  various  fragmentary 
and  elliptical  utterances  contextualized  as  answers  to  tutor  questions.  Many  issues  remain 
however,  including  more  effective  handling  of  conjunctions/compounds,  as  well  as  pronoun 
references  and  elisions.  Some  other  specific  features  identified  for  inclusion  in  Phase  II  include: 
(1)  feedback  to  the  student  to  clarify  the  system’s  interpretation  of  their  input;  (2)  on-demand 
hints  to  the  student  on  the  types  of  input  that  are  currently  expected;  and  (3)  spelling  correction 
on  inputs  to  minimize  misunderstandings  based  on  flawed  inputs. 

Multi-part  input  (multiple  sentences) 

One  important  aspect  of  the  prototype’s  free  text  input  capability  is  that  it  is  able  to  deal 
with  fairly  arbitrary  blocks  of  input  text.  In  particular  it  is  not  limited  to  one  well-formed 
sentence  at  a  time.  When  accepting  the  student’s  orders  to  his  battalion,  the  prototype  can  parse 
any  number  of  sentences  together,  including  sentences  with  more  than  one  phrase  (e.g.,  both  a 
maneuver  and  a  military  objective).  This  is  important  since  our  observations  of  students  indicate 
that  they  will  generate  utterances  of  vastly  differing  lengths  and  completeness.  Going  forward 
we  have  planned  extension  mechanisms  to  deal  with  even  more  radically  abbreviated  and 
extended  forms  of  input. 

Student  solution  gap-filling 

Student  input  can  come  in  different  sized  chunks,  and  that  implies  different  degrees  of 
completeness  after  one  or  several  inputs.  For  instance,  when  the  system  has  asked  the  student  to 
enter  a  set  of  orders,  it  has  an  idea  of  what  would  constitute  a  reasonably  complete  input.  So 
long  as  it  is  missing  items  it  expects,  it  will  prompt  the  user  to  fill  in  missing  pieces  at  the  end  of 
each  received  piece  of  input.  In  the  demonstrated  prototype,  this  is  illustrated  by  prompts  for 
orders  to  units  that  have  not  yet  been  given  any  instructions.  It  is  also  possible  to  prompt  for 
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missing  details  in  orders  to  a  particular  unit — for  instance  if  a  unit  is  told  to  move  somewhere, 
but  is  not  told  what  to  do  when  it  gets  there. 

Student  solution  classification 

Once  a  set  of  orders  are  input  as  text  and  verified  through  graphical  manipulations  the 
system  attempts  to  classify  the  student’s  proposed  response  to  the  scenario  situation.  For 
example  the  orders  shown  above  from  our  demonstration  sequence  can  reasonably  be  classified 
as  a  cautious  approach  to  the  situation.  The  student  is  not  running  away  from  the  enemy,  but 
neither  is  he  seizing  the  initiative  and  attacking  the  main  formation.  The  tutoring  that  ensues 
addresses  the  issues  of  time  and  distance  and  initiative.  In  particular,  what  is  going  to  happen 
when  the  enemy  scouts  get  to  the  top  of  the  hill  and  discover  elements  of  the  student’s  battalion? 
Contrast  this  with  the  following  alternate,  more  aggressive,  orders  that  the  student  may  enter  on  a 
later  cycle  through  the  solicit-or ders  —classify— Hut or  cycle: 

Alpha  go  to  Balzerton  and  observe.  I  also  want  my  tows  to  move  to  Balzerton.  Bravo  you 

move  to  the  top  of  Balzer  Hill  and  open  fire  on  enemy  tank  battalion.  Charlie  go  to  top  of 

hill  also  and  attack  the  enemy  tanks.  Delta  head  to  south  of  Balzer  Hill  and  observe. 

Here  the  student  is  planning  to  directly  take  on  the  main  enemy  formation.  The  system 
diagnoses  this  as  fitting  its  bold  case.  In  this  case  the  system  drills  on  how  the  user’s  immediate 
and  future  actions  relate  to  the  higher  commander’s  intent. 

Case  topic/theme  scripting 

As  noted,  the  two  cases  just  illustrated  lead  to  tutoring  on  somewhat  different  topics  or 
themes.  The  important  point  is  that  each  diagnostic  case  fragment  suggests  a  set  of  topics  worth 
covering.  The  cautious  case  leads  to  a  line  of  discussion  about  the  situation  and  the  enemy’s 
intent  that  includes  questions  such  as: 

What  do  you  think  is  at  Vietti ’s  Farm? 

What  are  they  doing  right  now? 

What  do  you  think  they  ’ll  do  when  they  ’re  finished  refueling? 

Where  is  the  front  line? 

This  line  of  questioning  culminates  in  a  recap  of  the  essential  facts:  “ So  we ’ve  got  an 
enemy  tank  battalion  refueling  deep  behind  our  front  line  and  you  think  they  might  be  going  on 
the  offensive .”  It  further  leads  to  discussion  of  time  and  distance  issues  with  questions  such  as: 

How  long  before  the  2  T-72s  see  you? 

How  far  can  your  M1A1  travel  in  5  minutes? 

Who  do  you  think  will  reach  the  top  of  the  hill  first? 

Where  will  your  battalion  be  when  the  T-72s  hit  the  top  of  the  hill? 

What  do  you  think  is  going  to  happen  [back  at  the  main  laager  site]  when  their  T-72s  up 

there  start  shooting  and  your  battalion  starts  returning  fire  to  the  top  of  the  hill? 
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Again  the  line  of  question  culminates  in  a  recap:  “So  at  this  point  it  seems  as  though  you 
have  lost  the  initiative  as  far  as  engaging  that  enemy  battalion .”  It  opens  the  way  to  revisiting 
the  initial  orders  generated  by  the  student: 

What  could  you  have  done  differently? 

Would  you  say  time  is  most  critical  factor  here? 

Do  you  want  to  try  again? 

General  scenario  tutoring  cycle 

The  prototype  illustrates  a  general  goal  structure  for  tutoring  a  scenario.  The  system  sets 
up  the  graphical  presentation  of  the  scenario  and  prints  the  initial  situation  description.  It  solicits 
the  student’s  solution  (a  set  of  orders),  and  verifies  those  orders  by  soliciting  updates  to  the 
situation  map.  It  classifies  the  student’s  solution  to  the  current  problem  and  tutors  on  a  set  of 
themes  that  follow  from  that  case/diagnosis.  It  potentially  offers  the  opportunity  to  cycle  back 
and  revise  the  orders,  leading  to  a  new  solution  classification  and  further  tutoring  on  newly 
relevant  themes.  It  may  move  on  through  a  series  of  decision  points  in  the  scenario,  each  with 
their  own  cycle  of  alternate  student  solutions.  Finally  the  scenario  session  ends  with  an  explicit 
opportunity  for  reflection.  Figure  5  illustrates  the  general  tutorial  session  goal  structure;  the 
italicized  lines  represent  processes  that  are  explicitly  part  of  the  model,  but  were  not 
implemented  in  the  Phase  I  prototype. 

Rim  scenario  session 
Set-up  map 
Introduce  scenario 
Check  scenario  clarity 
Prompt  for  decision 

Solicit  initial  orders 
Solicit  map  updates 
Classify  solution  case 

Check/solicit  rationale 
Tutor  on  verified  case 

Possible  loop  back  to  prompt  for  new  decision 
<Additional  choice-points  and  student  solution  cycles 
...> 

Wrap-up  session  (reflection) 


Figure  5.  ComMentor  General  Goal  Structure  for  Scenario  Tutoring  Session. 
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Tutor  goal  management 

In  addition  to  a  general  goal  structure  for  scenario-based  tutoring  sessions,  the  Phase  I 
prototype  also  illustrates  a  more  general  goal  management  mechanism.  The  system  maintains  a 
goal  agenda.  The  active  goal  is  responsible  for  generating  some  output  (generally  a  question)  to 
prompt  the  user  for  additional  input.  That  goal  will  also  have  a  set  of  expectations  to  be  used  in 
interpreting  whatever  input  is  received.  Depending  on  the  input  received  and  any  other  facts  that 
have  accumulated  in  the  systems  session  memory,  the  goal  may  lead  to  the  spawning  of  new 
goals,  or  may  rearrange  the  contents  of  the  goal  agenda.  All  pending  goals  can  have  pending 
expectations,  which  can  be  used  to  analyze  the  input  should  the  current  active  goal  not  be  able  to 
arrive  at  a  suitable  interpretation.  All  together  this  flexible  set  of  mechanisms  allows  for  (1) 
sequenced  behaviors,  (2)  branching  behaviors,  (3)  context-sensitive  behaviors,  (4)  varying  levels 
of  alternate  or  default  behaviors. 

Prompts  for  reflection 

As  suggested  in  “General  scenario  tutoring  cycle,”  the  last  part  of  any  scenario  session  is 
a  wrap-up  interaction  aimed  at  encouraging  reflection.  A  standard  set  of  questions  is  used  here 
including  the  following: 

What  do  you  think  you  did  well  in  this  situation? 

What  do  you  think  you  could  have  done  better? 

Going  forward,  the  main  issue  here  is  to  have  the  system  actually  check  student  input 
against  its  record  of  student  solution  diagnoses  generated  during  the  session.  However,  even  in 
the  absence  of  a  system-generated  reality  check,  a  simple  canned  question  sequence  begins  to 
serve  the  purpose  of  encouraging  student  reflection  on  the  lessons  of  the  session. 

Phase  II  System  Design 


Phase  II  Technical  Objectives 

During  Phase  II,  we  plan  first  to  extend  our  work  on  the  Phase  I  objectives: 

1 .  Continue  to  identify  and  encode  tactical  analysis  knowledge  that  should  be 
represented  by  the  system  to  support  the  evaluation  of  learner  solutions  to  tactical  scenarios,  in 
order  to  refine  the  system’s  student  model.  In  Phase  II,  particular  emphasis  will  be  placed  on  (a) 
getting  reasonably  complete  coverage  of  attributes  and  inferences  in  basic  areas  such  as  units 
vehicles,  and  weapons,  complemented  by  authoring  tools  to  ease  additions  and  modifications  to 
the  accumulated  knowledge-base;  (b)  characterization  and  prioritization  of  the  space  of  possible 
extension  sub-domains  of  tactical  analysis  knowledge;  and  (c)  further  exploration  of  ways  to 
unify  ComMentor’s  tactical  analysis  knowledge  with  other  efforts  in  the  Department  of  Defense 
(DoD)  community  (e.g.,  adoption  of  proposed  knowledge  representation  language  standards  and 
evolving  ontologies  such  as  those  sponsored  in  DARPA  research  programs  like  high 
performance  knowledge  base  (HPKB)  or  command  post  of  the  future  [CPOF]). 
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2.  Continue  to  explore  and  evaluate  candidate  interaction  techniques  that  achieve  the 
same  types  of  instructional  goals  achieved  by  learner-mentor  interactions  carried  out  by  expert 
human  instructors.  In  Phase  II  we  will  devote  particular  emphasis  to  (a)  exploring  the 
extensibility  of  our  current  textual  language  processing  approach  versus  other  more  structured 
input  techniques;  (b)  exploring  the  applicability  of  evolving  techniques  for  speech  input  and 
output;  (c)  pushing  on  the  utility  of,  and  coordination  techniques  for,  multi-modal  interactions 
that  combine  graphics  and  text  (or  speech)  input  and  output. 

3.  Continue  to  identify,  classify,  and  implement  the  types  of  tutoring  strategies  that 
can  be  automated  in  order  to  select  and  generate  effective  instructional  interventions  based  on 
the  student  model.  Our  Phase  I  work  illustrates  application  of  some  of  the  tutoring  strategies 
already  identified  (e.g.,  see  Table  1  and  accompanying  discussion).  Many  of  the  identified 
strategies  remain  to  be  implemented  or  even  seriously  investigated.  Others  likely  remain  to  be 
discovered.  Our  psychology  subcontractor  is  particularly  eager,  and  well  positioned  to  develop  a 
model  of  Tutoring  Instruction/Mentoring  Strategies.  Additional  data  collection  and  preliminary 
development  of  such  a  theory  during  our  Phase  II  effort  should  help  to  provide  a  firmer  empirical 
and  theoretical  foundation  for  ongoing  design  and  implementation  efforts. 

4.  Refine  our  methods  for  authoring  ITS  scenarios  by  entering  good  and  bad  sets  of 
decisions  and  rationale,  and  then  annotating  these  solutions  with  skills  and  knowledge  that  are 
demonstrated  (positively  or  negatively)  by  each  solution.  In  Phase  II,  emphasis  will  be  placed  on 
systematizing  our  approach  to  these  tasks,  developing  custom  authoring  tools,  and  embedded 
guidance  in  application  of  those  tools.  Efforts  to  ease  acquisition  of  this  scenario-specific 
information  complement  the  efforts  in  objective  1  to  get  a  handle  on  the  general  (domain- 
specific)  tactical  analysis  knowledge. 

5.  Develop  a  fully  functional  software  prototype  that  not  only  illustrates  the  key  ideas 
related  to  tactical  analysis  knowledge  representation,  learner-mentor  interaction  methods,  and 
interactive  tutoring  strategies,  but  also  serves  as  an  effective  learning  aid  for  Army  officers.  We 
expect  to  observe,  analyze  and  implement  on  the  order  of  another  six  scenarios  during  Phase  II. 

In  concert  with  all  the  objectives  listed  above,  this  is  a  logical  next  step  in  efforts  to  scale-up  the 
techniques  identified  and  illustrated  in  Phase  I. 

In  addition  to  the  above-stated  extensions  of  the  Phase  I  objectives,  we  plan  to  address 
the  following  new  Phase  II  objectives: 

6.  Demonstrate  the  effectiveness  of  the  tutoring  system  prototype  developed  in  Phase  II. 
Evaluation  will  be  an  ongoing  effort  throughout  the  Phase  II,  starting  with  formative  evaluations 
of  the  Phase  I  prototype,  continuing  through  early  user  trials  and  observations  of  the  evolving 
Phase  II  system,  and  culminating  in  a  controlled  study  of  system  effectiveness  using  tactical 
reasoning  skill  evaluation  metrics  and  instruments,  most  likely  adapted  from  those  developed  by 
ARI  for  use  in  the  TLAC  program. 

7.  Develop  guidelines  and  baseline  data  on  scenario  authoring.  As  a  result  of  developing 
a  half-dozen  scenarios,  we  will  acquire  a  far  better  grasp  of  effective  techniques,  typical  pitfalls, 
and  likely  costs  involved  in  extending  the  system.  In  addition  to  authoring  techniques 
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concretized  in  authoring  tools,  and  guidelines  explicitly  embedded  in  such  tools’  interfaces 
and/or  help  systems,  there  is  likely  to  be  other  “wisdom”  gained  about  how  to  build  effective 
scenarios,  including  the  costs  and  typical  tradeoffs  inherent  in  such  efforts.  The  results  of 
objectives  6  and  7  together  will  feed  into  addressing  our  final  objective  number  8. 

8.  Develop  a  preliminary  analysis  of  the  issues  likely  to  bear  on  long-term  acceptance, 
effectiveness,  and  maintainability  of  an  expanded,  fully  operational  ComMentor.  In  addition, 
consider  the  portability  of  the  techniques  embedded  in  ComMentor  to  other,  more  commercial 
domains  and  applications. 


Phase  II  Architecture 

Figure  6  shows  the  architecture  of  the  proposed  ComMentor  system,  broken  down  at  the 
top  level  into  (1)  the  ITS  User  Interface,  (2)  the  ITS  Core  Processes,  (3)  the  ITS  Authoring 
Tools,  and  (4)  a  set  of  persistent  knowledge  bases.  The  main  knowledge  bases  reflect  and 
support  much  of  the  underlying  structure  of  the  system.  For  instance  four  distinct  components  of 
the  Authoring  Tool  suite  address  each  of  the  four  major  types  of  knowledge:  Domain 
Knowledge,  Scenario  Knowledge,  Curricular  Knowledge,  and  Instructional  Control  Knowledge 
(or  strategies).  Likewise,  those  kinds  of  knowledge  (as  well  as  the  individualized  student  model) 
enter  into  the  main  ITS  operations  primarily  through  distinct  processing  modules. 

One  major  effect  of  this  design  is  to  simplify  knowledge  acquisition,  as  subject  matter 
experts  will  ultimately  spend  most  of  their  time  entering  tactical  analyses  for  each  scenario,  and 
will  not  need  to  concern  themselves  as  frequently  or  deeply  with  other  more  stable  sorts  of 
knowledge  (for  instance,  determining  how  the  newly  entered  scenario  knowledge  will  be  used). 

A  related  effect,  then,  is  to  allow  for  more  flexible  use  of  the  tactical  analyses,  as  the  Case 
Assessment  Module  can  employ  a  wide  range  of  algorithms  for  comparing  the  student's  solution 
with  those  of  the  experts  to  find  significant  similarities  and  differences,  and  the  Tutor  Interaction 
Module  can  likewise  use  a  wide  range  of  algorithms  to  select  instructional  interventions 
accordingly.  Finally,  this  kind  of  separation  allows  the  system  to  evolve  more  gracefully  to 
incorporate  new  and  improved  technologies,  by  supporting  piece-wise  replacement  of  the 
software  modules.  For  example,  in  Phase  II  we  intend  to  explore  modification  of  the  User 
Interface  by  introduction  of  a  Speech  Output  Generator  and  Speech  Input  Analyzer,  and  expect 
minimal  impact  on  the  other  major  modules  and  knowledge  bases. 
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Figure  6.  ComMentor  Phase  II  System  Architecture. 

Phase  II  Components 

The  ITS  User  Interface 

The  ComMentor  User  Interface  supports  all  interactions  with  the  user.  Specifically,  as 
illustrated  in  Figure  4  and  Figure  6,  this  module: 

•  Presents  tactical  scenarios  to  the  learner  in  the  form  of  a  map  display  plus  narrative. 

•  Accepts  solutions  from  the  learner  in  the  form  of  texts  (orders,  reports,  requests, 
augmented  with  tutorial  questions  and  answers),  supplemented  by  map  manipulations 
(placement  of  icons,  and  annotation  with  regions  and  paths). 

•  Enables  an  interactive  multimodal  instructional  dialog  that  combines  graphical  and 
textual  input  and  output,  while  embodying  a  range  of  pedagogical  strategies. 

The  above  is  a  fair  characterization  of  the  capabilities  illustrated  by  the  Phase  I  prototype. 
In  addition,  during  Phase  II,  we  intend  to  experiment  with  other  forms  of  interaction.  As  noted 
earlier,  the  most  likely  candidates  are: 

•  Presentation  of  war  stories  that  illustrate  lessons  related  to  the  learner's  solution  and/or 
reasoning. 

•  Extension  of  multimodal  capabilities  to  include  speech  input  and  output. 
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The  ITS  Core  Processes  and  Knowledge  Bases 

The  core  ITS  shown  in  Figure  6  elaborates  on  the  abstract  version  of  the  architecture 
originally  shown  in  Figure  1 .  The  basic  pipeline  structure — from  the  input  modules,  to  the 
Domain  Inference  Module  (DIM),  to  the  Case  Assessment  Module  (CAM),  to  the  Tutor 
Interaction  Module — is  preserved.  Remember,  the  idea  is  that  successive  modules  raise  the  level 
of  abstraction  in  the  situation  description  to  arrive  at  a  useful  tutoring-relevant  characterization. 
For  example,  the  input  module  might  be  able  to  generate  the  characterization:  “ Alpha  tank 
company  is  north  of  Balzer  Hill;  Bravo  and  Charlie  companies  are  on  top  of  Balzer  Hill;  Delta 
company  is  south  of  Baker  hill."  The  DIM  might  elaborate:  “ Alpha  and  Delta  companies  are 
not  in  range  to  attack  or  be  attacked  by  the  enemy  tank  formation,  and  Delta  company  cannot  be 
seen  by  that  formation;  Bravo  and  Charlie  companies  can  attack  the  enemy  formation ."  The 
CAM  might  say:  “ The  student  might  be  proposing  to  use  some  tank  companies,  located  on  the 
key  terrain  of  Balzer  Hill,  to  attack  the  enemy  tank  formation;  if  so,  that  is  a  recognized  solution, 
and  it  suggests  some  specific  tutoring  interactions ...  (e.g.,  verification  of plausible  rationale)." 
Figure  6  unpacks  some  additional  details,  primarily  related  to  the  knowledge  bases  that  support 
the  processing  modules’  activities. 

For  instance,  Figure  6  calls  out  in  detail  how  the  Expert  Model  is  composed  of  combined 
information  from  the  Domain  Knowledge  Base  and  the  Scenario  Knowledge  Base,  and  how  that 
combined  expert  model  supports  both  the  Domain  Inference  Module  and  the  Case  Assessment 
Module.  In  addition.  Figure  6  shows  that  the  Expert  Model  contributes  lexical/conceptual 
information  that  configures  the  Text  Input  Analyzer  to  enable  parsing  of  user  utterances. 

The  Domain  Knowledge  Base  contributes  baseline  facts  such  as  “ an  Ml  A I  has  an 
effective  range  of  2km"  or  “ the  top  speed  of  a  T-7 2  over  open  country  is  60  kmh "  and  rules  such 
as  “if  an  enemy  target  is  in  range  of  a  weapon,  then  it  can  be  attacked  using  that  weapon."  It 
may  also  contain  tactically  relevant  knowledge  about  how  such  facts  can  be  modified  by 
mitigating  factors  such  as  night  operations,  crew  training,  weather,  etc.  In  such  cases,  capturing 
possible  ranges,  and  the  contextual  parameters  they  depend  on  is  a  useful  technique.  Over  time 
the  Domain  Knowledge  Base  can  potentially  be  extended  to  support  such  forms  of  reasoning  as 
trafficability  analysis,  in  order  to  make  more  nuanced  predictions  about  how  long  certain 
battlefield  maneuvers  might  take. 

The  Scenario  Knowledge  Base  contributes  facts  such  as  “Blue  has  a  tank  battalion, 
composed  of  four  companies  with  14  M1A1  tanks  apiece ”  and  “Baker  Hill  is  2km  from  Vietti ’s 
Farm."  It  also  contains  diagnostic  case  fragments  that  can  recognize  “ the  student  positioning 
tanks  on  Balzer  Hill  is  evidence  in  favor  of  an  aggressive  attack  on  the  enemy  position ”  and  can 
recommend  partial  tutoring  scripts  such  as  “(1)  verify  the  student’s  situation  assessment  and 
rationale,  then  (2)  dwell  on  time-sensitive  aspects  of  (a)  the  tactical  plan,  (b)  how  the  orders 
were  issued  communicating  that  plan,  and  (c)  how  the  situation  was  reported  to  higher 
command,  then  (3)  transition  to  discuss  the  longer-term  situation  assessment  and  possible 
plans."  These  suggestions  can  be  passed  on  to  the  Tutor  Interaction  Module. 
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The  Tutor  Interaction  Module  gets  an  understanding  of  possible  issues  and  ways  of 
teaching  about  them  from  the  Curriculum  Knowledge  Base.  For  instance,  one  curricular  point 
might  be:  “ Environmental  issues  put  constraints  on  how  effective  orders  will  be .”  The 
Curriculum  Knowledge  Base  should  know  that  ToF  is  one  scenario  that  can  be  used  to  raise  this 
point,  and  that  in  that  scenario,  the  relevant  environmental  conditions  are  alertness  of  crews 
based  on  time  of  day  and  visibility,  current  position  and  activity.  It  should  know  that  there  are  a 
range  of  general  and  specific  points  to  make  about  this  issue,  including,  potentially,  anecdotes 
about  how  experienced  officers  have  coped  with  such  situations. 

One  likely  way  to  incorporate  lesson-bearing  personal  anecdotes  into  the  system  is  to 
support  capture,  storage,  and  playback  of  desktop  video  recordings.  While  this  clearly  would 
require  augmenting  ComMentor  authoring  tools,  from  the  perspective  of  our  current  focus  on  the 
knowledge  bases,  the  interesting  new  requirements  have  to  do  with  story  indexing.  Indexing  is 
the  problem  of  annotating  stories  with  descriptors  so  that  they  can  be  found  when  they  are 
relevant  to  interactions  in  the  tutoring  system.  Attaching  stories  to  curricular  elements  is  a 
plausible  start  at  indexing,  but  some  stories  can  be  used  in  more  than  one  context,  while  other 
stories  only  make  sense  when  some  very  specific  set  of  circumstances  are  met.  Indexing  is  thus 
one  of  the  key  issues  to  be  explored  as  we  investigate  the  possible  utility  of  incorporating  war 
stories  into  ComMentor. 

The  Tutor  Interaction  Module  also  gets  guidance  about  the  application  and  sequencing  of 
pedagogical  techniques  from  the  Instructional  Strategies  Knowledge  Base.  This  knowledge 
base  can  contain  general  sequencing  suggestions  such  as  “(1)  bring  student  to  understanding  of  a 
potential  problem,  (2)  solicit  possible  solutions  (3)  evaluate  and  highlight  shortcoming  of 
solution  (4)  iterate  on  solution,  and  finally  (5)  offer  model  solution.  ”  It  can  also  contain  topic 
nomination  rules  such  as:  "If  the  learner  made  a  decision  D1  in  situation  S,  and  an  expert 
solution  for  situation  S  contains  a  decision  D2,  a  relevant  discussion  topic  is  whether  D1  or  D2 
is  a  better  response  to  situation  S. " 

Finally,  Figure  6  introduces  two  feedback  paths  from  the  Tutorial  Interaction  Module 
back  to  the  DIM.  The  first  path  goes  through  the  Session  Model,  which  maintains  a  history  of 
what  has  gone  on  in  the  current  session.  It  knows  what  topics  have  been  covered,  what  issues 
have  been  raised,  what  solutions  have  been  attempted,  what  instructional  interventions  have  been 
executed,  and  what  responses  have  been  received.  This  recent  memory  must  inform  the 
interpretation  of  student  inputs  (and  down  through  the  pipeline,  must  affect  situation  assessment 
and  tutorial  actions). 

The  second  feedback  path  goes  through  the  Student  Model,  which  maintains  a  longer- 
term  view  of  what  the  student  is  believed  to  know  and  not  know.  For  instance,  it  records  in  a 
persistent  form  what  other  scenarios  the  student  has  been  through,  what  curricular  topics  were 
covered  (and  how  they  were  covered,  e.g.,  were  relevant  expert  stories  presented?),  and  to  what 
extent  mastery  of  those  topics  was  demonstrated.  This  information  is  critical  not  only  to  biasing 
assessments  of  student  performance  during  a  scenario,  but  also  to  helping  the  system  choose 
which  scenarios  would  be  of  value  to  the  particular  student. 
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ComMentor  Authoring  Tools 

We  plan  to  develop  an  authoring  tool  for  each  of  the  four  main  knowledge  bases  as 
pictured  in  Figure  6.  The  Student  Model  does  not  need  an  authoring  tool,  as  it’s  content  is 
entirely  under  program  control,  based  primarily  on  the  content  of  the  Curriculum  Knowledge 
Base,  with  contributions  from  other  knowledge  bases  as  well.  We  will  focus  on  the  Scenario 
Authoring  Tool  initially,  with  a  secondary  emphasis  on  the  Curriculum  Authoring  Tool.  Over 
time,  effort  will  shift  to  the  Instructional  Strategies  Tool.  The  Domain  Authoring  Tool  will  be 
de-emphasized;  for  the  most  part,  we  expect  to  provide  simple  functionality  for  defining 
categories  and  filling  in  object  templates  (e.g.,  vehicles  and  weapons). 

The  Scenario  Authoring  Tool  will  start  by  requiring  a  basic  scenario  specification, 
including  a  scenario  name,  map,  task  organizations  (with  any  idiosyncratic  supporting  icons), 
and  introductory  text(s).  This  basic  information  provides  enough  context  for  the  domain  expert 
to  begin  specifying  possible  student  configurations  and  sequences — the  good  and  bad  cases  that 
will  be  used  to  help  evaluate  students’  proposed  solutions  when  running  the  scenario.  Following 
the  lead  of  other  tactical  decision  ITS  projects  at  SHAI  each  such  case  can  be  constructed  using 
an  interface  essentially  equivalent  to  the  basic  ComMentor  student  interface. 

Once  constructed,  however,  these  cases  must  be  evaluated,  generalized,  and  annotated. 
The  evaluations  indicate  the  strong  and  weak  points  of  the  proposed  partial  student  solution. 
Generalizations  allow  the  expert  to  indicate  some  aspects  of  the  case  pattern  that  may  be  varied 
while  still  retaining  the  essential  nature  (and  evaluation)  of  the  situation4.  Annotations  include 
citations  of  domain  facts  and  higher-level  situation  assessments  that  play  into  justifying  the 
evaluation  of  the  situation  pattern.  Among  the  annotations  may  be  references  to  elements  drawn 
from  the  Curriculum  Knowledge  Base — that  is  the  expert  may  say  that  a  given  case  fragment 
illustrates  (or  contravenes)  a  curricular  principle.  Finally,  specific  tutoring  sketchy  scripts — 
(partially  specified)  sequences  of  tutoring  actions — can  be  tied  to  these  cases. 

The  Curriculum  Authoring  Tool  allows  the  domain  expert  to  define  curricular  points  (or 
principles)  of  the  sort  that  can  be  referenced  by  individual  scenarios  and  their  diagnostic  cases. 

As  is  typical  of  SHAI  ITS’s,  the  curriculum  can  be  structured  to  indicate  dependencies  (e.g., 
part/whole  relationships  as  well  as  logical  dependencies).  For  example,  a  plausible  curricular 
principle  is  that  you  should  keep  your  higher  commander  informed  of  important  developments  of 
which  he  may  be  unaware.  Immediately,  there  arise  the  questions  of  what  counts  as  an  important 
development,  and  what  constitutes  evidence  that  the  higher  commander  may  be  unaware  of 
something.  A  good  understanding  of  the  first  principle  clearly  depends,  logically,  on  an 
understanding  of  the  later  two  principles.  As  with  the  scenario  cases,  curricular  elements  can 
also  have  tutoring  scripts  attached.  One  particularly  important  kind  of  tutoring  information 
likely  to  be  attached  here  are  the  war  stories  referred  to  earlier. 

The  Instructional  Strategies  Authoring  Tool  is  the  user  interface  for  specifying 
generalized  tutoring  scripts  and  the  recognition  conditions  that  license  their  application.  The 
basic  idea  is  to  pair  sketchy  tutoring  scripts,  with  student  and  session  conditions.  As  already 


4  Certain  kinds  of  case  sequence  generalization  were  explored  and  implemented  as  a  core  technique  in  SHAI’s  Task 
Tutor  Toolkit. 
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suggested,  some  tutoring  strategies  can  be  triggered  more  or  less  directly  on  scenario  conditions 
(e.g.,  case  recognition),  or  on  curricular  conditions  (e.g.,  the  need  to  tutor  on  a  particular  point). 
Authoring  of  such  strategies  will  be  tightly  bound  to  their  contexts,  and  specification  of  the 
trigger  conditions  largely  handled  implicitly.  But  we  believe  there  will  also  be  more  general 
triggers  for  more  general  strategies — as  suggested,  for  instance  by  Jona  (1995).  Work  on  this 
tool  will  dovetail  with  the  important  project  research  focus  on  identifying  and  implementing 
Socratic  tutoring  strategies  (e.g.,  Task  4  in  “Phase  II  Tasks  and  Milestones”). 

Finally,  the  Domain  Knowledge  Authoring  Tool  is  likely  to  be  largely  limited  to 
elaboration  of  conceptual  taxonomies  and  the  filling  in  of  forms  pre-designed  to  cover  important 
classes  of  domain  entities.  For  instance,  a  vehicle  form  might  provide  places  to  enter  a  class 
(from  a  pre-existing  taxonomy),  a  range  before  refueling,  a  capacity  (for  people  and  cargo),  a 
max  speed  (perhaps  different  over  different  classes  of  terrain),  and  so  on.  Exact  schemas  for 
classes  like  vehicles,  weapons,  etc.  may  plausibly  be  derived  from  existing  sources  such  as  the 
Military  Intelligence  Data  Base  (MIDB).  We  note  again,  however,  that  ranges  on  such  values 
(parameterized  by  relevant  contextual  factors)  are  often  more  useful  and  appropriate  to  support 
tactical  reasoning. 

One  additional  point  remains  to  be  made  about  the  enterprise  of  authoring,  and  the 
construction  of  these  authoring  tools.  Stottler  Henke  Associates,  Inc.  has  extensive  experience 
with  this  problem,  and  an  evolving  tool  set;  however,  the  direct  applicability  of  prior  work  varies 
for  the  different  kinds  of  knowledge.  While  curriculum  authoring  is  a  relatively  settled  matter, 
what  it  takes  to  author  appropriate  scenarios  and  domain  knowledge  for  a  given  ITS  (or  for  that 
matter  any  AI  system)  is,  in  part,  learned  through  experience  with  early  examples,  and  then 
through  additional  experience  with  early  authoring  tool  users.  The  authoring  of  Socratic  tutoring 
strategies  as  already  noted,  is  still  a  significant  research  problem,  but  one  that  SHAI,  in 
combination  with  our  subcontractor,  is  particularly  well  positioned  to  address.  We  have 
structured  our  Phase  II  work  so  that  we  will  get  early  and  frequent  experience  in  both  scenario 
encoding,  and  domain  expert  interaction  with  authoring  tools;  we  plan  four  waves  of  authoring 
tool  implementation  and  observation.  One  concrete  payoff  of  such  experiences  is  that  lessons 
learned  can  be  rolled  into  the  authoring  tools,  either  through  changes  in  interfaces,  or  through 
tailored  help  systems  (or  even  special  authoring  tool  training  systems!). 

Future  Work 


Phase  II  Plans 

Phase  II  Tasks  and  Milestones.  We  propose  the  following  1 6  tasks  to  achieve  the  Phase 
II  objectives. 

1 .  Perform  Formative  Evaluation  of  Phase  I  Prototype 

We  will  begin  our  Phase  II  work  by  conducting  a  more  complete  formative  evaluation  of 
the  Phase  I  prototype  than  was  possible  within  the  scope  of  the  Phase  I  project  itself.  This 
evaluation  should  include  hands-on  sessions  with  the  system,  both  for  our  consulting  subject 
matter  experts,  as  well  as  for  more  junior  military  personnel  of  the  type  who  would  constitute  an 
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appropriate  user  trainee  audience.  The  object  is  to  observe  the  sessions  looking  for  aspects  of  the 
system  that  seem  to  work  well,  as  well  as  for  flaws  and  limitations.  We  will  also  collect  the 
users’  comments  and  opinions,  and  feed  those  into  our  ongoing  design  and  implementation 
efforts. 


2.  Identify  and  Adapt  Phase  II  Scenarios  and  Plan  Tutorial  Observation  Sessions 

Building  on  our  successful  approach  to  data  collection  on  naturalistic  battlefield 
reasoning  tutoring,  developed  during  Phase  I,  we  will  work  with  our  consultants  and 
subcontractors  to  select  a  set  of  six  tactical  scenarios,  and  arrange  tutorial  sessions  with  military 
personnel  as  students  and  our  consultants  as  expert  instructors.  We  expect  to  observe 
approximately  four  tutoring  sessions  per  scenario.  The  bulk  of  these  will  be  with  our  subject 
matter  expert  consultants  acting  as  instructors.  However  to  broaden  our  base  of  observations,  we 
will  also  intend  to  run  some  sessions  with  alternate  instructors.  In  those  cases,  we  expect  to  have 
our  consultants  act  as  observers,  as  a  way  of  gaining  additional  insight  into  the  session  conduct, 
and  as  a  way  to  encourage  reflection  on  the  ways  they  conduct  their  own  sessions.  The  selection 
of  scenarios  and  generation  of  a  session  and  observation  plan  will  constitute  the  first  project 
milestone,  to  be  completed  by  the  end  of  Month  2. 

3.  Carry  out  Tutorial  Observation  Sessions  and  Analyze  Results 

We  expect  that  actual  tutoring  sessions  will  be  conducted  using  the  distributed  approach 
piloted  so  successfully  in  our  Phase  I  work.  Members  of  both  the  project  teams  will  participate 
as  observers.  We  expect  to  begin  these  sessions  perhaps  as  early  as  the  second  month  of  the 
project,  and  project  that  all  tutorial  sessions  and  observations  will  be  completed  before  the  end  of 
year  one  of  the  contract. 

4.  Develop  Instructional  Strategies  Model 

One  of  the  major  outputs  of  the  tutoring  session  analysis  will  be  an  Instructional 
Strategies  Model.  The  goal  is  to  establish  a  solid  empirical  and  theoretical  basis  for  the 
Instructional  Strategies  Knowledge  Base.  We  expect  the  final  draft  will  be  produced  following 
the  final  set  of  observed  tutoring  sessions.  Generation  of  the  final  Instructional  Strategies  Model 
will  constitute  a  project  milestone,  to  be  completed  by  the  end  of  Month  13. 

5.  Elaborate  Instructional  Strategies  Knowledge  Base 

Elaboration  of  the  Instructional  Strategies  Knowledge  Base  will  occur  in  waves  (as  will 
be  true  for  the  other  three  knowledge  based  as  well).  The  first  two  passes  at  augmenting  this 
knowledge  base  will  follow  completion  of  the  two  drafts  of  the  Instructional  Strategies  Model. 
Remember,  the  purpose  of  the  Model  in  the  context  of  this  project  is  to  help  us  systematically 
identify  and  organize  our  tutoring  control  knowledge.  We  have  also  reserved  a  block  of  time  and 
effort  towards  the  end  of  the  project  (in  Months  19-21)  during  which  we  can  perform  a  final 
reorganization  and  extension  of  this  knowledge  base  (and  in  fact  all  the  knowledge  bases)  based 
on  what  we  have  learned  during  the  first  %  of  the  project,  in  preparation  for  the  final  release  and 
evaluation  of  the  Phase  II  system. 
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6.  Elaborate  Scenario  Knowledge  Base 

The  development  of  scenario-specific  knowledge  is  a  top  priority  in  this  project.  The 
central  contents  of  the  scenario  knowledge  base  will  be  fragmentary  scenario  solutions 
representing  good  and  bad  approaches  to  the  problem.  Each  such  fragment  will  include  a  set  of 
recognition  conditions,  as  well  as  a  set  of  belief  and  rationale  statements  with  indications  of  how 
those  statements  relate  to  the  facts  of  the  scenario,  to  good  domain  practice,  and  to  possible 
tutoring  moves.  Blocks  of  time  and  effort  have  been  scheduled  for  this  task  following 
completion  of  the  three  waves  of  tutoring  session  observations.  As  noted  later,  the  second  and 
third  of  these  blocks  will  coincide  with  early  efforts  at  developing  end-user  authoring  tools,  and 
will  serve  to  guide  those  efforts  (also  note  that  exercises  in  the  use  of  the  developing  authoring 
tools  will  follow  immediately  on  the  heels  of  work  on  scenario  knowledge  base  extension  and 
authoring  tool  construction).  As  with  all  the  knowledge  bases,  there  is  time  reserved  in  Months 
19-21  for  a  final  cleanup  and  overhaul  prior  to  completing  the  final  Phase  II  system  release. 

7.  Elaborate  Curricular  Knowledge  Base 

This  final  knowledge  base  contains  information  about  the  principles,  concepts,  and  skills 
to  be  taught  by  the  ITS.  That  information  includes  interrelationships  among  curricular  items,  as 
well  as  possible  ways  to  present  information  about,  or  probe  for  understanding  of  them. 
Ultimately,  the  individual  student  model  is  formed,  in  large  part,  as  a  reflection  of  this  curricular 
knowledge  base — recording  what  topics  the  student  has  been  exposed  to,  in  what  form,  and  with 
what  success.  Work  on  elaborating  the  curriculum  will  come  in  waves,  the  first  three  following 
on  efforts  devoted  to  extending  the  scenario  knowledge  base.  Of  course,  as  usual,  one  extra 
block  of  time  is  reserved  for  a  final  cleanup  and  overhaul  prior  to  completing  the  final  Phase  II 
system  release. 

8.  Elaborate  Domain  Knowledge  Base 

It  is  critical  to  have  a  basic  competence  for  representing  and  reasoning  about  the  facts  of 
the  battlespace,  but  we  intend  to  limit  the  effort  devoted  to  stretching  that  competence.  That  is 
because  the  development  of  formal  reasoning  models  for  such  a  complex  domain  is  known  to  be 
an  open-ended  (and  quite  difficult)  problem;  also,  other  research  programs  have  focused,  and  are 
focusing,  on  developing  such  competences.  From  experience  in  developing  other  case-based 
ITSs,  we  believe  the  greatest  payoff — in  rapid  system  construction  and  end-user  extensibility — 
will  be  achieved  by  focusing  relatively  more  effort  on  scenario-specific  knowledge.  That  said, 
we  plan  to  devote  some  effort  early  on  (during  Months  4  and  5)  to  getting  our  basic  domain 
model  established.  We  will  also  devote  effort  to  this  task  as  we  approach  completion  of  the 
second  major  release  of  the  Phase  II  ComMentor  (during  Months  16  and  17).  Finally,  as  with  all 
the  knowledge  bases,  we  have  reserved  a  block  of  time  before  completing  the  third  and  final 
major  release  of  the  Phase  II  ComMentor  system  to  one  last  reorganization  and  extension  of  the 
system’s  domain  knowledge. 


34 


9.  Refine  Phase  II  System  Design 

From  a  primary  concern  with  data  collection  and  knowledge  codification,  we  shift  here  to 
system  design  and  implementation.  Early  in  the  project,  based  in  part  on  the  initial  Phase  I 
prototype  formative  evaluation,  we  will  review  the  proposed  Phase  II  system  design  and  look  for 
newly  uncovered  requirements,  or  opportunities  to  improve  the  system.  The  ultimate  result  of 
this  effort  will  be  a  Phase  II  design  document  that  will  guide  subsequent  implementation  efforts. 
Generation  of  this  document  will  constitute  a  milestone  for  the  project,  to  be  completed  by  the 
end  of  Month  5. 

10.  Implement  ITS  Core  Extensions 

We  propose  to  devote  a  substantial  block  of  implementation  time  early  in  the  project  to 
enhance  and  refine  the  basic  system  code  originally  developed  for  the  Phase  I  prototype,  in 
keeping  with  the  dictates  of  the  newly  refined  Phase  II  system  design.  The  object  is  to  provide  a 
firm  foundation  for  a  more  functional,  scalable  and  robust  system.  This  overhauled  system  base, 
combined  with  expanded  knowledge  structures  and  a  first  pass  the  authoring  tools  will  constitute 
the  first  Phase  II  release  of  ComMentor. 

1 1 .  Explore  Alternate  Interaction  Mechanisms 

As  part  of  the  work  on  the  second  Phase  II  ComMentor  release,  we  propose  to 
experiment  with  a  range  of  alternate  interaction  mechanisms.  Two  major  examples  cited  above 
include:  (1)  implementing  a  capability  to  capture  expert  anecdotal  “war  stories,”  and  then 
effectively  deploy  them  during  tutoring  sessions;  and  (2)  implementing  a  capability  to  manage 
tutorial  input  and  output  in  the  form  of  speech.  For  the  “war  stories”  effort,  we  intend  to  draw  on 
earlier  work,  primarily  at  Northwestern’s  Institute  for  the  Learning  Sciences,  on  conversational 
video  archives.  For  the  speech  input/output  work  we  intend  to  draw  on  ongoing  efforts  at 
SHAI’s  Seattle  office  on  robust  speech  processing  for  training  applications. 

The  introduction  of  new  interaction  mechanisms  such  as  these  has  implications  for  the 
authoring  process,  and  we  expect  to  see  that  reflected  in  the  later  waves  of  authoring  tool 
development.  We  class  these  extensions  as  experimental  because  they  require  substantial 
implementation  (and  ultimately  authoring)  effort,  and  there  exists,  as  yet,  relatively  little 
experience  with  how  to  most  effectively  integrate  such  capabilities  with  ComMentor’ s  style  of 
tutoring.  Achieving  adequate  performance,  appropriate  forms  of  integration,  and  efficient 
authoring  constitute  technical  research  topics 

12.  Implement  Authoring  Tools 

The  development  of  end-user  authoring  tools  will  constitute  a  significant  thrust  of  this 
project  and  will  be  spread  over  all  three  releases  of  the  ComMentor  system.  Early  on,  we  expect 
to  focus  on  helping  to  capture  scenario  information.  Over  time,  we  will  aim  to  extend  the 
authoring  tool’s  coverage  to  include  curricular  knowledge  (including  war  stories),  and 
instructional  strategies  as  well.  Our  domain  expert  consultants  will  contribute  to  this  effort 
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through  their  comments  on  proposed  design,  and  through  participation  in  the  authoring  tool 
application  exercises  described  in  the  next  task. 

13.  Observe  subject  matter  expert  (SME)  Use  of  Authoring  Tools 

Following  each  wave  of  authoring  tool  construction  we  will  devote  a  slice  of  time  to 
observation  and  reflection  on  how  those  tools  are  used  by  our  consulting  domain  experts.  The 
idea  is  to  ensure  that  the  tools  we  build  are  usable  by  their  intended  audience,  and  to  leam  what 
is  working  and  not  working  as  quickly  as  possible.  The  final  set  of  these  observations  is 
expected  to  occur  just  before  we  begin  our  final  evaluation  studies.  Our  intent  is  to  have  our 
domain  experts  make  an  effort  to  enter  one  or  two  new  scenarios  using  the  final  versions  of  the 
authoring  tools,  and  to  include  these  new  scenarios  in  the  final  evaluation  to  compare  their 
effectiveness  to  the  other,  more  painstakingly  analyzed  and  encoded  scenarios. 

14.  Evaluate  Phase  II  System  Releases 

In  keeping  with  our  iterative  development  approach,  we  aim  to  interleave  evaluation  (and 
possible  course  corrections)  into  all  the  cycles  of  our  work.  To  mark  the  end  of  the  first  two 
ComMentor  release  cycles  we  plan  to  perform  interim  evaluations  of  the  system.  These 
evaluations  will  involve  use  by  representatives  of  the  system’s  intended  audience.  Initially 
formative  in  character,  the  evaluations  will  transition  to  summative,  culminating  in  a  final  formal 
evaluation  of  the  third  release.  The  two  interim  evaluations  under  this  Task  will  serve  as 
milestones,  marking  the  completion  of  release  cycles  1  and  2  by  the  end  of  Months  10  and  17 
respectively. 

To  mark  the  end  of  the  final  ComMentor  release  cycle,  we  will  perform  a  full,  formal 
evaluation  of  the  system,  as  used  by  representative  members  of  its  intended  user  population. 
Planning  for  this  evaluation  will  begin  as  early  as  Month  4.  We  expect  to  adapt  and  apply  the 
battlefield  reasoning  evaluation  metrics  we  understand  to  be  under  development  by  ARI  in 
support  of  TLAC.  Completion  of  this  study  by  the  end  of  Month  23  will  mark  the  project’s 
penultimate  milestone. 

15.  Analyze  System’s  Long-Term  Deployment  Issues 

In  order  to  provide  as  complete  a  picture  as  possible  of  ComMentor’ s  utility,  we  propose 
to  devote  time  at  the  very  end  of  the  project  to  identifying,  and,  as  far  as  is  possible,  quantifying, 
the  likely  costs  and  benefits  of  deploying  the  system  and  maintaining  it  over  the  long  haul.  The 
object  is  to  get  a  clear  vision  of  the  road  to  Phase  III  adoption  of  and  wide-spread  use  of  this 
work. 


16.  Prepare  Final  Technical  Report 

In  the  last  month  of  the  project  we  will  write  up  our  results  in  a  final  technical  report, 
documenting  our  efforts  and  accomplishments.  Completion  of  this  report  by  the  end  of  Month 
24  will  mark  the  final  project  milestone. 
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Phase  III  Transitional  Plan 


At  the  end  of  Phase  II,  we  expect  to  have  developed: 

•  A  catalog  of  instructional  strategies  that  can  be  employed  to  support  Socratic  interactions 
for  battlefield  command  reasoning  and  for  other  similar  learning  domains. 

•  A  knowledge  representation  design  for  tactical  scenario  facts,  scenarios,  and  analysis  that 
can  be  used  to  support  an  intelligent  tutoring  system. 

•  A  scenario-based  intelligent  Socratic  tutoring  system  that  provides  mentor-student 
interactions  to  teach  battlefield  command  reasoning  skills. 

•  An  application-specific  authoring  tool  emphasizing  creation  by  subject  matter  experts  of 
new  teaching  scenarios  comprised  of  tactical  scenarios  along  with  "good"  and  "bad" 
solutions. 

The  Phase  II  research  and  development  will  provide  a  solid  foundation  for  Phase  III 
commercialization  in  four  ways.  First,  we  will  have  a  reasonably  complete  catalog  of  Socratic 
tutoring  strategies,  and  experience  implementing  many  specific  tutoring  techniques.  Second  we 
will  have  experience  building  and  evaluating  a  complete  Socratic  tutor  aimed  at  high  level 
reasoning  skills — a  first  in  the  field.  Third,  we  will  have  a  working  system  that  actually  teaches 
battlefield  reasoning  skills  using  a  limited  number  of  cases,  and  the  authoring  tools  to  help  grow 
the  system  cost-effectively.  Finally,  we  expect  to  have  the  interest  and  attention  of  relevant  part 
of  the  DoD  training  community.  We  believe  our  Phase  II  system  will  be  compelling  to  a  wide 
range  of  potential  customers,  not  the  least  of  which  would  be  training  organizations  within  DoD. 

Possible  Markets  for  High-Level  Socratic  Tutors 

The  proposed  research  presents  enormous  commercialization  potential.  Web-based 
training  offers  the  promise  of  providing  training  "any-time,  any-place."  However,  current  web- 
based  training  technologies  frequently  focus  on  teaching  facts  and  procedures  rather  than 
reasoning  skills  and  thinking  habits.  We  believe  that  enhanced  ITS  technologies  that  teach 
reasoning  skills  will  be  of  tremendous  value  to  both  government  and  commercial  customers. 

Consider  a  key  example,  already  introduced  above.  Stottler  Henke  Associates,  Inc.  is 
currently  working  with  Simulation,  Training  and  Instrumentation  Command  (STRICOM)  to 
develop  the  command,  control,  communications,  computers,  intelligent  tutoring  system  (C4ITS) 
for  the  Armor  Captain  Career  Course  (a  course,  incidentally,  open  to  many  officers  who  are  not 
strictly  Armor  Captains).  We  expect  the  reasoning  skills  orientation  and  Socratic  tutoring 
capabilities  of  ComMentor  to  form  a  tremendously  useful  adjunct  to  the  more  traditional 
scenario-enactment  training  orientation  of  C4ITS.  We  believe  that  a  combined  system  would 
find  a  very  receptive  audience,  and  hope  for  a  clear  path  to  fielding  a  full-scale  implementation. 

In  general,  we  plan  to  use  the  reasoning  skills  ITS  technology  developed  during  this 
project  to  offer  the  following  software  products  and  services: 

•  Technology-enabled  consulting  services  to  help  web-based  education  service  providers 
offer  reasoning  skills  tutoring  systems.  Potential  targets  include  providers  of  paper-based 
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case  teaching  materials  who  are  starting  to  offer  software  tutors  (e.g.,  Harvard  Business 
School  Press),  and  on-line  providers  of  learning  software  for  K-12,  university,  and 
graduate  school-level  programs. 

•  Technology-enabled  consulting  services  that  deliver  turnkey  ITS  applications  for  large, 
end-user  companies  with  sophisticated  training  needs. 

•  Tutoring  system  engines  and  authoring  tools  to  consulting  companies  who  provide 
sophisticated  training  solutions  to  large  companies.  For  example,  Andersen  Consulting 
applies  authoring  tools  developed  by  its  partner,  the  Institute  for  the  Learning  Sciences, 
to  develop  intelligent  tutoring  systems  for  its  clients. 

SHAI’s  Market  Development  Capabilities 

We  have  identified  intelligent  tutoring  systems  as  a  strategic  direction  for  the  company. 
Although  many  companies  are  using  the  Web  to  facilitate  communications  among  students  and 
educators  or  to  simplify  access  to  traditional,  electronic  course  materials,  no  companies  yet  offer 
or  provide  a  vision  for  web-enabled  intelligent  tutoring  systems.  We  believe  that  intelligent 
tutoring  systems  are  essential  for  enabling  web-based  learning  systems  to  achieve  the  vision  of 
helping  students  learn  "any  time,  any  place."  To  position  ourselves  for  this  important  market  we 
are  undertaking  much  of  our  development  effort  using  the  Web-friendly  Java  programming 
language. 

We  will  focus  our  marketing  efforts  on  consulting  services  provided  to  educational 
software  providers,  consulting  companies,  and  early-adopter  end  user  companies.  We  will  sell 
these  services  via  direct  consultative  sales  to  identify  custom  software  solutions  that  support 
those  customer's  unique  needs.  These  sales  activities  will  be  supported  by  a  combination  of 
marketing  and  lead-generation  activities  including  direct  mail,  presentations  at  educational 
industry  events,  and  articles.  For  example,  during  the  year  2000,  SHAI  delivered  presentations 
describing  intelligent  tutoring  systems  technologies  and  benefits  at  the  Interservice/Industry, 
Simulation  and  Education  Conference  (I/ITSEC)  2000  (three  presentations),  American  Society 
for  Training  and  Development  (ASTD)  TechKnowledge  2000  conference,  and  the  SmartSystems 
conference  sponsored  by  NASA  and  other  organizations.  During  the  year  2000,  we  wrote  (or 
helped  write)  articles  that  appeared  in  the  ASTD  LeamingCircuits  electronic  magazine  for 
technical  training  (http://www.leamingcircuits.  org/feb2000/ong.html)  and  the  San  Jose  Mercury 
News. 


The  STRICOM  relationship  highlighted  above  is  an  example  of  a  marketing  opportunity 
that  could  be  an  early  route  to  turning  the  results  of  this  research  and  development  project  into  a 
production  system.  Similar  positive  relationships  with  the  Navy’s  Surface  Warfare  Officers 
School  (a  legacy  of  the  tactical  action  officer  [TAO]-ITS  project)  could  serve  a  similar  end. 

Both  would  be  examples  of  technology-enabled  consulting  that  delivers  turnkey  ITS  applications 
for  end-users  with  sophisticated  training  needs. 

Through  government-funded  and  commercial  software  development  projects,  SHAI  has 
amassed  significant  expertise  and  software  assets,  which  make  us  technologically  competitive 
with  the  small  number  of  potential  ITS  competitors.  We  currently  have  on-going  software 
consulting  relationships  with  major  educational  software/content  vendors,  and  we  are  confident 
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of  our  ability  to  establish  new  relationships  with  a  wider  range  of  software/content  vendors  in 
other  educational  markets  to  disseminate  SHAI's  ITS  technology  through  established  (or  rapidly 
growing)  product  vendors.  Likewise,  we  have  a  growing  network  of  satisfied  customers 
throughout  the  DoD  world. 

Stottler  Henke  Associates,  Inc.  has  the  resources  and  expertise  to  support  the  sales  and 
business  development  effort  required  to  pursue  direct  sales  of  technology  by  enabling  consulting 
projects  and  licensing  arrangements  to  end-user  companies,  other  vendors,  and  consulting 
companies.  Stottler  Henke  Associates,  Inc.  continues  to  augment  its  business  development  staff 
and  budget  to  exploit  commercial  product  and  consulting  opportunities.  For  example,  James 
Ong  serves  in  engineering  management  and  business  development  roles  at  SHAI.  Before 
coming  to  SHAI,  Mr.  Ong  was  the  director  of  product  marketing  at  Belmont  Research,  Inc.  and 
led  the  commercialization  of  the  TableTrans®  data  transformation  system,  developed  with 
partial  funding  from  the  National  Institute  of  Standards  and  Technology  (NIST)  Advanced 
Technology  Program  (project  94-04-0024:  "Voyager:  Browsing  and  Automatically  Extracting 
Healthcare  Data  from  Scattered  Databases").  TableTrans  was  selected  by  NIST  as  a  NIST/ATP 
success  story  (http://www.atp.nist.gov/atp/success/belmont.htm)  and  is  co-marketed  by  IBM  to 
pharmaceutical  companies  to  streamline  clinical  data  handling. 

Conclusions 

During  this  Phase  I  SBIR  project  we  accomplished  several  important  goals.  First,  we 
developed  a  deep  understanding  of  the  battlefield  command  domain,  and  of  existing  techniques 
for  training  battlefield  command  reasoning  skills.  What  we  learned  reinforced  the  importance  of 
making  Socratic  tutoring  more  widely  available  to  commanders-in-training,  and  suggested  that 
the  TDG  format  was  an  effective  context  for  such  training.  We  identified  a  range  of  levels  at 
which  such  tutoring  could  be  conducted,  and  we  experienced  first-hand  the  importance  of  getting 
good  data  on  which  to  base  the  design  of  an  automated  tutor. 

Driven  by  this  initial  survey,  we  conducted  six  sample  tutorial  observations.  These 
observations  served  as  the  basis  for  developing  an  initial  taxonomy  of  tutorial  moves.  Our 
taxonomy  encompassed  scenario -based  Socratic  tutoring  moves,  general  topic  management 
moves,  specific  theme-related  tutoring  moves,  and  a  variety  of  tutoring  moves  associated  with 
styles  of  tutoring  other  than  what  is  classically  considered  “Socratic.” 

Based  on  our  domain  understanding,  and  our  tutoring  observations  and  analysis,  we 
designed  and  built  a  limited  proof-of-concept  prototype  of  the  ComMentor  automated  Socratic 
tutoring  system.  The  point  of  the  prototyping  effort  was  to  help  advance  our  design  ideas 
through  experimentation,  and  to  demonstrate  some  of  the  key  ideas  and  technologies  that  would 
go  into  a  complete  Phase  II  prototype.  We  demonstrated  a  capability  for  multi-modal  input  and 
output  (graphics  and  text)  including  parsing  of  moderately  complex  and  variable  natural 
language  orders.  We  demonstrated  an  initial  capability  for  the  tutoring  system  to  pursue  gaps  in 
student  inputs,  and  to  classify  proposed  student  problem  solutions  into  categories  that  could  drive 
the  tutoring  session  forward  by  working  through  thematically  scripted  interactions  that 
nonetheless  maintained  sensitivity  to  the  student’s  inputs.  All  of  this  was  built  to  function  in  the 
context  of  a  tutor  goal-management  mechanism,  and  a  general  scenario-based  tutoring  cycle  that 
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showed  how  techniques  other  than  Socratic  tutoring  could  be  combined  (e.g.,  closing  prompts 
intended  to  facilitate  student  reflection). 

Finally,  we  incorporated  lessons  learned  from  our  Phase  I  prototyping  effort  into  a  project 
plan  and  system  design  for  a  Phase  II  effort.  We  believe  we  have  developed  a  system  design  and 
a  project  work  plan  that  will  use  the  available  resources  to  significantly  advance  the  state  of  the 
art  with  respect  to  automatically  tutoring  high-level  command  reasoning  skills.  That  plan 
includes  full-scale  prototype  implementation  of  the  ComMentor  system  with  approximately  six 
scenarios,  and  an  accompanying  suite  of  authoring  tools.  It  also  includes  evaluation  studies  to 
assess  how  well  the  system  functions.  We  believe  our  Phase  II  plan,  as  well  as  our  other 
corporate  experiences  and  contacts,  will  position  us  to  take  the  results  of  a  Phase  II  effort  on  to 
successful  Phase  III  commercialization.  The  technologies  to  be  developed  here  will  provide  a 
significant  increase  in  the  capability  of  ITSs,  while  lowering  the  costs  of  fielding  such  systems. 
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Appendix  A 
Related  Work 
ITS  Research  at  SHAI 


AEGIS  Tactical  Action  Officer  ITS 

Stottler  Henke  Associates,  Inc.  has  developed  for  the  U.S.  Navy  a  simulation-based 
Intelligent  Tutoring  System  (ITS)  which  enables  students  to  act  as  tactical  action  officer  (TAOs) 
in  tactical  simulations.  The  simulation’s  graphical  user  interface  displays  a  geographical  map  of 
the  region  and  provides  rapid  access  to  sensor,  weapon,  and  communication  functions.  After  the 
student  completes  a  scenario,  the  ITS  evaluates  the  entire  sequence  of  student  actions  to  infer 
tactical  principles  that  the  student  correctly  applied  or  failed  to  apply.  These  principles  are 
detected  according  to  sophisticated  pattern-matching  algorithms  defined  by  the  instructor  using 
the  System’s  graphical  user  interface.  The  system  is  highly  configurable  within  the  domain  of 
naval  tactical  simulations,  and  authoring  tools  enable  the  instructor  to  define  new  types  of  ships 
and  aircraft,  scenarios,  and  principles.  The  instructor  can  also  define  complex  behaviors  for  each 
friendly  and  enemy  ship  and  aircraft  to  create  realistic,  intelligent,  multi-agent  simulations.  The 
TAO  ITS  has  proven  an  effective  training  tool.  Tactical  action  officer  students  at  the  Surface 
Warfare  Officers  School  are  currently  using  it,  and  initial  independent  evaluation  of  the  software 
in  use  has  been  highly  favorable.  Simulation-based  intelligent  training  systems  complement 
traditional  classroom  or  computer-based  training  by  enabling  students  to  practice  the  application 
of  concepts  and  principles.  Additional  funding  has  been  received  to  adapt  the  TAO  ITS  for  fleet 
use  on  board  ships.  Contact:  Joe  Russell  (703)  602-5959  xl 83. 

C 41  ITS 

Under  contract  to  STRICOM,  SHAI  is  currently  developing  C4I  ITS,  a  prototype 
intelligent  tutoring  system  for  armored  and  mechanized  infantry  company  commanders.  The  C4I 
ITS  will  teach  tactical  decision-making,  command  and  control  principles,  and  the  use  of  the 
Force  XXI  Battle  Command,  Brigade  and  Below  (FBCB2)  command  and  control  system.  Before 
each  mission,  students  will  issue  pre-mission  orders  and  graphical  overlays  that  specify 
movements.  The  tutoring  system  will  assess  these  plans  using  symbolic  pattern  recognition 
techniques  that  compare  each  student's  plan  with  annotated  good  and  bad  plans  (or  portions  of 
plans)  supplied  by  experts.  Similarities  between  the  student's  plan  and  the  good  plans  will 
identify  specific  proficiencies  in  high-level  and  low-level  skills.  Similarities  between  the 
student's  plan  and  bad  plans  will  identify  skill  deficiencies. 

During  the  mission,  the  students  will  interact  with  FBCB2  and  Spearhead,  a  military 
game  simulation  program  that  is  being  adapted  for  use  in  military  training  and  is  being  integrated 
with  FBCB2.  Spearhead  implements  the  simulation  behaviors  and  presents  a  3  dimensional  "out 
the  window"  view  of  the  world  as  seen  from  each  vehicle.  Spearhead  will  exchange  real-time 
simulation  data  with  FBCB2,  and  C4I  ITS  will  intercept  those  packets.  The  tutoring  system  will 
construct  scenarios  and  modify  scenarios  in  progress  to  include  situations  that  test  various 
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situation  assessment  skills.  For  example,  the  tutoring  system  could  place  friendly  forces  in 
certain  locations  that  the  student  might  otherwise  fire  upon.  If  the  student  does  fire  upon  those 
positions,  the  tutoring  system  can  infer  that  the  student  failed  to  use  FBCB2  to  notice  the 
presence  of  friendly  forces  in  that  area.  As  another  example,  the  tutoring  system  could  place 
enemy  forces  at  a  location  such  that  the  student  should  direct  friendly  forces  to  oppose  them. 
Failure  to  perform  this  action  indicates  either  a  lack  of  situation  awareness  of  the  enemy  forces, 
or  a  poor  tactical  decision.  Contact:  Rodney  Long  at  STRICOM,  (407)  384-3928. 

Internet  Intelligent  Tutoring  System  Authoring  Tool 

In  connection  with  a  project  to  make  ITS  systems  more  easily  distributed  over  the 
Internet,  as  well  as  allowing  separate  ITS  systems  to  interoperate,  SHAI  is  developing  an 
authoring  tool — IITSAT — that  addresses  several  of  the  authoring  tool  needs  identified  for 
ComMentor.  In  particular  IITSAT  provides  for  the  authoring  of  curriculum  knowledge  as 
hierarchies  of  principles  to  be  taught,  and  that  can  be  interrelated,  in  addition,  by  prerequisite 
annotations.  In  IITSAT,  principles  have  multimedia  courseware  attached,  and  can  be  linked  to 
scenarios  that  rely  on  those  principles,  as  well  as  to  specific  critiques  that  determine  when  a 
principle  is  being  followed  or  violated.  In  ComMentor,  curricular  elements  will  be  linked  to 
scenarios  and  cases  (critiques),  but  instead  of  multimedia  courseware,  we  will  be  concerned  with 
alternate  forms  of  instruction,  such  as  tutoring  moves  and  expert  stories.  IITSAT  provides  an 
API  to  plug  in  custom  scenario  authoring  tools  (since  scenario  structure  vary  radically  across 
domains),  and  provides  a  first  instance  of  a  critique  authoring  tool.  The  ComMentors  scenarios, 
with  their  good  and  bad  cases  will  fill  these  niches.  Finally,  IITSAT  contains  an  Instructional 
Methods  authoring  component.  However,  unlike  ComMentor’s  proposed  Socratic  methods, 
IITSAT  focuses  more  on  simpler  techniques  to  adapt  instruction  to  individual  students  (e.g.,  how 
many  exercises  to  give  on  any  given  topic,  how  quickly  to  give  hints,  how  many,  and  what  type 
of  examples  to  provide,  etc.). 

Speech  Interaction  for  ITSs 

Stottler  Henke  Associates,  Inc.  has  been  tasked  by  the  Navy,  in  a  project  sponsored  by 
the  Office  of  the  Secretary  of  Defense,  to  develop  an  innovative  system  that  will  advance  the 
state-of-the-art  in  computer  application  support  for  natural  speech  dialogue.  The  design  of  this 
system  ties  together  the  best  of  the  current  research  in  speech  recognition,  natural  language 
understanding,  and  dialogue  modeling,  together  with  an  eclectic  combination  of  supporting 
Artificial  Intelligence  techniques,  into  a  complete  end-to-end  design.  In  our  Training 
Application  Language  Kit  (TALK)  architecture,  all  components  in  the  natural  speech  dialogue 
system  will  provide  disambiguation  and  validation  for  the  output  of  the  others.  In  addition,  all 
components  will  operate  using  a  shared  set  of  resources,  including  a  representation  of  the  current 
dialogue  context,  generic  and  domain-specific  dictionaries,  and  dialogue  models.  Sharing  these 
resources  between  the  components  of  the  TALK  model  not  only  ensures  consistency  in  the 
interpretation  and  generation  of  natural  speech  dialogue,  but  also  provides  an  unprecedented 
capability  for  training  the  system  for  new  domains  and  integrating  dialogue  capability  into  a 
wide  range  of  applications,  such  as  instructional  systems  and  automated  customer  support. 

Phase  I  of  this  Small  Business  Innovation  Research  (SBIR)  program  began  on  January  26,  2001. 
Client:  NAWCTSD,  Orlando,  FL.  Contact:  Steve  Slosser,  (407)  380-4599. 
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Dismounted  Infantry  Military  Operations  in  Urban  Terrain  (MOUT)  Intelligent  Tutoring  System 

Stottler  Henke  Associates,  Inc.  developed,  in  cooperation  with  Research  Development 
Corporation,  a  simulation-based  ITS  (SITS)  for  training  dismounted  infantry,  both  as  individuals 
and  teams,  that  would  be  used  with  a  virtual  reality  simulator.  Included  in  the  project  was  the 
development  of  a  generic  automated  training  system  (ATS)  architecture  that  can  interface  with 
existing  and  future  simulators.  The  SITS  diagnoses  student  learning  needs,  determines  what 
instruction  content  and  technology  are  most  appropriate,  and  drives  the  presentation  of  that 
instruction.  Key  technologies  are  case-based  reasoning  (CBR),  integrated  knowledge  structures 
for  representing  expert  and  student  knowledge,  automatic  knowledge  elicitation,  and  dynamic 
scenario  selection  and  creation.  The  architecture  supports  automatic  and  semi-automatic 
knowledge  engineering  to  update  its  knowledge  base  as  the  domain  itself  evolves.  The  system 
trains  squad  and  fire  team  leaders  in  Military  Operations  in  Urban  Terrain.  The  ATS  monitors 
the  student's  actions  in  the  virtual  reality  environment,  assesses  his  deficiencies,  and  modifies  the 
scenario  or  creates  new  ones  to  address  those  deficiencies.  A  CBR  system  also  selects  the  most 
appropriate  instructional  technique  based  on  the  student's  individual  requirements  and  past 
learning  behavior. 

Constructivist  Distance  Learning  System  for  Counter-Terrorist  Intelligent  Analysis 

Stottler  Henke  Associates,  Inc.  is  developing  for  the  U.S.  Army  at  Fort  Huachuca  a 
training  system  comprised  of  two  parts.  The  first  is  a  general  framework  that  supports  the 
creation  of  Constructivist  distance  learning  (DL)  courseware  in  a  wide  variety  of  areas.  The 
second  product  is  the  Intelligence  in  Combating  Terrorism  (ICT)  Tutor,  a  specific  tutoring  and 
scenario  authoring  system  built  using  this  general  course  creation  framework. 

The  purpose  of  the  ICT  Tutor  is  to  give  students  extensive,  hands-on  training  in  the 
analysis  of  raw  intelligence  information  about  terrorist  organizations  and  installation  threat 
assessments,  leading  to  a  compact  summary  of  a  terrorist  operation  and  an  assessment  of  the 
current  level  of  threat.  The  tutor  uses  Constructivist  learning  theory  by  supporting  adaptive 
learning,  modeling,  intentional  activity,  and  rich  scenario  contexts.  Contact:  Helen  Remily  at 
(520)  533-9077. 

Task  Tutor  Toolkit  and  Remote  Payload  Operations  Tutor  for  Procedural  Training 

To  lower  the  cost  and  difficulty  of  creating  scenario-based  intelligent  tutoring  systems  for 
procedural  task  training,  SHAI  developed  the  Task  Tutor  Toolkit  (T3),  a  generic  tutoring  system 
shell  and  scenario  authoring  tool.  The  Task  Tutor  Toolkit  employs  a  case-based  reasoning 
approach  where  the  instructor  creates  a  procedure  template  that  specifies  the  range  of  student 
actions  that  are  "correct"  within  each  scenario.  The  system  enables  a  non-programmer  to  specify 
task  knowledge  quickly  and  easily  via  graphical  user  interface,  using  a  "demonstrate,  generalize, 
and  annotate"  paradigm  that  recognizes  the  range  of  possible  valid  actions  and  infers  general 
principles  that  are  understood  (or  misunderstood)  by  the  student  when  those  actions  are  carried 
out.  The  annotated  procedure  template  also  enables  the  Task  Tutor  Toolkit  to  provide  hints 
requested  by  the  student  during  scenarios,  such  as  “What  do  I  do  now?”  and  “Why  do  I  do  that?” 
At  the  end  of  each  scenario,  Remote  Payload  Operations  Tutor  (RPOT)  displays  the  principles 
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correctly  or  incorrectly  demonstrated  by  the  student,  along  with  explanations  and  background 
information.  The  Task  Tutor  Toolkit  was  designed  to  be  modular  and  general  so  that  it  can  be 
interfaced  with  a  wide  range  of  training  simulators  and  support  a  variety  of  training  domains. 

Stottler  Henke  Associates,  Inc.  and  NASA  used  the  Task  Tutor  Toolkit  to  create  the 
RPOT,  a  tutoring  system  application  which  lets  scientists  who  are  new  to  space  mission 
operations  learn  to  monitor  and  control  their  experiments  aboard  the  International  Space  Station 
according  to  NASA  payload  regulations,  guidelines,  and  procedures.  Contact:  Mr.  Stephen 
Noneman,  (256)  544-2048.  Phase  II  Completed:  February  2000. 

Intelligent  Tutoring  System  for  Long-Range  Acoustic  Detection  of  Submarines 

Stottler  Henke  Associates,  Inc.  is  developing  for  the  U.S.  Navy  an  Acoustic  Analysis 
Intelligent  Tutoring  System  (AAITS)  which  will  enable  students  to  practice  the  detection  and 
classification  of  sources  of  underwater  acoustic  signals  such  as  submarines  and  whales.  Acoustic 
analysis  experts  will  create  scenarios  using  a  Scenario  Authoring  Tool  by  selecting  and  viewing 
LOFARGRAMs  which  are  frequency-analyzed  acoustic  datasets  displayed  as  2D  images, 
annotating  them  with  significant  features  and  links  among  related  features,  providing  reasons  for 
requesting  each  LOFARGRAM,  and  assigning  a  final  classification.  Students  will  use  the 
Tutoring  System  to  carry  out  this  same  acoustic  analysis.  By  comparing  the  details  of  each 
student's  analysis  with  those  of  the  expert,  the  Tutoring  System  can  identify  the  acoustic  analysis 
principles  understood  and  correctly  applied  by  each  student,  provide  specific  and  individualized 
feedback,  suggest  relevant  training  materials,  and  select  appropriate  next  scenarios.  By  storing 
LOFARGRAMs  annotated  by  experts,  AAITS  also  serves  as  a  knowledge  repository  which 
disseminates  the  most  current  acoustic  analysis  expertise  to  sonar  technicians  on  land  or  at  sea. 

A  key  innovation  of  AAITS  is  the  use  of  an  application-specific  Scenario  Authoring  Tool  that 
enables  experts  to  create  scenarios,  which  encode  their  expertise  and  analyses  intuitively,  by 
annotating  datasets  graphically,  using  a  point-and-click  graphical  user  interface.  Phase  II  project 
start  date:  February  1999.  Contact:  Master  Chief  Joseph  Spivey  at  SPAWAR,  (858)  537-0312. 

AI  Representations  of  Military  Plans  and  Tactics  at  SHAI 

Stottler  Henke  Associates,  Inc.  has  developed  several  systems  which  employ  artificial 
intelligence  representations  of  military  plans  and  tactics.  Selected  projects  are  described  below. 

Intelligent  Control  for  Immersive  Wargaming 

Stottler  Henke  Associates,  Inc.  is  developing  a  wargaming  system  to  fuse  entity-level 
control  with  aggregate-level  command  to  manage  a  small-scale  air  campaign.  The  Phase  I 
prototype  does  this  by  combining  an  entity-level  robotic  control  architecture  called  Hap,  based 
on  research  at  Carnegie  Mellon  University,  with  a  strategic  command  system.  The  system 
allows  simulation  participants  to  interact  with  intelligent  allies  and  enemies.  For  example,  if 
there  is  a  goal  to  destroy  an  enemy  airbase,  the  aggregate-level  command  will  generate  plans  to 
form  a  strike  package.  Entities  controlling  the  elements  (fighters,  ECM,  and  ground  attack) 
coordinate  together  to  rendezvous  outside  enemy  airspace,  form  the  strike  package,  and 
overcome  enemy  resistance  in  a  team-oriented  fashion.  This  current  project  was  started  in  May 
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1999  and  the  contact  is  David  Ross  at  Air  France  Research  Labs  (AFRL)  Rome  Laboratories  - 
(315)330-7624. 

Multi-User  Simulation  Environment  (MUSE) 

Our  Phase  I  prototype  dealt  with  the  need  to  intelligently  control  all  the  tactical  platforms 
in  a  scenario.  We  developed  a  schema  for  military  warfare  tactics  representation  and  execution 
in  MUSE  using  expert  system  and  graphical  techniques  to  capture  the  human  decision  processes. 
Agents  in  the  scenario  would  compute  intercept  paths  to  engage  enemy  aircraft.  The  networked 
prototype  ran  with  geographically  distributed  team  players,  both  human  and  artificial.  Phase  I 
was  completed  in  December  of  1998  and  the  contact  is  Terry  Jackson  at  AFRL  Brooks  AFB  - 
(210)  536-3908. 

Robustness  in  Air  Campaign  Plans 

One  important  attribute  of  air  campaign  plans  is  robustness.  How  successful  will  a  plan’s 
execution  be  in  the  presence  of  uncertainty  and  changes,  and  how  hard  is  it  to  change  the  plan 
once  execution  has  begun?  Military  planners  can  gauge  the  robustness  of  air  campaign  plans. 
Furthermore,  plans  produced  by  automatic  planners  tend  to  be  graded  poor  on  this  scale.  A  team 
of  cognitive  psychologists  worked  to  unravel  and  understand  the  thought  processes  involved. 
Stottler  Henke  Associates,  Inc.  was  tasked  with  analyzing  this  abstract  body  of  work,  developing 
computable  concepts  which  parallel  these  processes,  implementing  them,  and  interfacing  them  to 
a  planning  system  in  current  use.  This  work  involved  an  understanding  of  the  entire  planning 
process  from  the  highest  level  to  the  most  detailed.  Client:  Klein  Associates.  Contact:  Dr.  Tom 
Miller,  513-873-8166.  Completion  Date:  September  1997. 

Related  Work  by  Others 
Mixed  Initiative  and  Socratic  Tutoring  Systems 

Research  in  mixed-initiative  and  Socratic  tutoring  systems  has  been  carried  out  for  more 
than  30  years.  In  fact,  much  of  the  early  work  in  intelligent  tutoring  systems  focused  on  the 
support  of  dialogs  between  the  software  and  the  student.  Early  work  includes  the  Scholar  system 
that  taught  South  American  geography  (Carbonell,  1970).  It  is  significant  in  its  early  use  of 
domain  knowledge  representations  (about  geography)  that  are  separate  from  the  instructional 
decision-making  knowledge.  However,  this  system  taught  facts  about  geography  rather  than 
reasoning  skills  and  therefore  did  not  pursue  many  goals  typical  of  Socratic  dialog.  Collins 
(1976)  developed  a  set  of  two  dozen  decision-making  rules  used  by  Socratic  tutors  and  embodied 
these  rules  within  the  WHY  system  that  taught  meteorological  reasoning  about  rainfall 
processes.  Clancey  (1987)  developed  GUIDON,  a  tutoring  system  for  diagnosing  blood  diseases 
that  used  as  its  expert  model  the  Mycin  rule-based  expert  system.  A  finding  of  the  GUIDON 
project  was  that  expert  system  rule-bases  frequently  employed  reasoning  methods  (in  this  case, 
rules)  that  were  not  intuitive  when  used  to  generate  tutorial  dialog  utterances.  Edelson  (1992) 
describes  Socratic  tutoring  in  the  domain  of  biology  that  relies  on  a  case-base  for  its  expert 
model. 
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This  previous  work  provides  useful  findings  and  insights,  especially  in  the  identification 
of  tutoring  strategies,  and  we  expect  to  build  upon  this  prior  work.  However,  ComMentor  will 
require  advancements  in  the  areas  of  using  complex  plans  and  rationale  structures  supplied  by 
experts  as  the  basis  for  tutoring  dialogs;  using  multiple  expert  solutions  for  assessing  student 
solutions;  and  assessing  and  enhancing  high-level  thinking  skills  and  habits  as  opposed  to 
detailed  knowledge  and  lower-level  reasoning  skills. 

Intelligent  Tutoring  System  Authoring  Tools 

The  ITS  technology  has  been  under  development  for  over  three  decades  by  researchers  in 
education,  psychology,  and  artificial  intelligence  at  military  and  academic  research  labs. 
Research  studies  carried  out  so  far  show  that  students  taught  using  ITSs  generally  performed 
better  and  learned  faster,  compared  to  classroom-trained  students.  However,  a  major 
impediment  to  the  widespread  use  of  intelligent  tutoring  systems  is  that  they  frequently  require 
custom  software  development  and  are  therefore  expensive  to  create.  Consequently,  the 
development  of  authoring  tools  to  enable  lower-cost  development  of  these  tutoring  systems  is 
also  an  area  of  active  research. 

Much  of  the  prior  research  in  ITS  authoring  tools  has  been  in  support  of  procedural  task 
training  domains  where  there  is  a  "correct"  method  for  carrying  out  each  procedure.  Guralnik 
(1996)  describes  an  authoring  tool  which  applies  a  content  theory  of  task  knowledge  which 
enables  the  tutoring  system  to  generate  replies  to  important  questions  from  the  student,  such  as 
"What  do  I  do  next?"  and  "How  do  I  do  that?"  (Munro  &  Pizzini,  1995)  describes  the  RIDES 
system,  which  enables  authors  to  create  graphical  training  simulations  (frequently,  of  devices) 
integrated  with  the  intelligent  tutor. 

The  ComMentor  will  require  a  new  type  of  ITS  authoring  tool,  one  that  captures  the 
reasoning  of  experts  in  support  of  solutions,  even  when  there  is  no  single  correct  solution.  We 
expect  that  ComMentor  may  draw  upon  prior  research  in  ITS  authoring  tools,  but  ComMentor 
will  require  significant  advances  in  the  capture  and  application  of  knowledge  structures  that 
capture  the  expert's  scenario-specific  reasoning. 

Goal-based  Scenarios,  Problem-solving  Architectures,  Generic  Tasks 

Learning  different  types  of  problem-solving  tasks  will  warrant  different  types  of  tutoring 
system  architectures.  However,  we  believe  that  there  are  a  manageable  number  of  distinct  types 
of  abstract  problem-solving  tasks.  We  are  encouraged  and  inspired  by  research  in  Generic  Tasks 
(Chandrasekaran,  1987),  Components  of  Expertise  (Steels,  1990),  and  Goal -based  Scenarios 
(Schank,  Fano,  Bell  &  Jona,  1994),  who  all  argue  that  the  number  of  types  of  problem-solving 
skills  is  quite  finite.  Proponents  of  Goal-based  Scenarios  at  the  Institute  for  the  Learning 
Sciences  (ILS)  at  Northwestern  University  assert  that: 

•  Students  learn  effectively  when  they  are  pursuing  goals  within  the  simulated  scenarios, 

and. 
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•  One  can  categorize  goals  into  a  small  number  of  abstract  types  (e.g.,  Investigate  and 
Decide,  Run  a  System  or  Organization,  Pursuade,  etc.)  which  strongly  affect  the 
architecture  of  the  appropriate  tutoring  system. 

Our  proposed  project  team  includes  Dr.  Eric  Domeshek,  a  project  manager  at  SHAI. 
Before  coming  to  SHAI,  Dr.  Domeshek  taught  and  carried  out  research  at  ILS  where  he 
developed  a  pair  of  successfully  fielded  educational  systems  for  university  courses,  as  well  as 
several  other  AI  and  multimedia  systems  for  knowledge  management.  One  of  these  systems, 
named  "Invitation  to  a  Revolution,"  implements  a  Goal-based  Scenario  (Pursuade)  to  help 
students  understand  the  social  forces  that  shaped  France  just  prior  to  the  Revolution.  The 
software  includes  a  hypermedia  database  of  over  290  video  clips  (over  4.5  hours  total)  of 
experts,  as  well  as  229  clips  of  12  different  historical  characters  which  support  simulated 
conversations  between  the  student  and  those  characters. 
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Appendix  B 

“Tanks  On  the  Farm”  Sample  Session  Transcript 
Problem 

The  following  table  represents  a  transcript  from  one  of  our  Phase  I  instructional 
observations.  The  first  column  of  the  table  contains  timing  information  and  comments  inserted 
by  the  session  analyst.  The  second  and  third  columns  represent  the  text  of  utterances  made  by 
the  Instructor  and  Student  respectively.  Column  four  is  a  first  pass  at  categorizing  the  type  of 
interaction  contained  in  a  given  row  of  the  table.  Column  five  is  a  similar  attempt  to  discern 
what  triggered  a  given  utterance  by  the  Instructor.  Columns  six  and  seven  tag  each  instructor 
utterance  according  to  the  instructional  theme  being  addressed  (based  originally  on  the  TLAC 
themes),  and  what  level  of  scaffolding  is  being  exhibited  (based  originally  on  the  TLAC-related 
scaffolding  levels  discussed  in  the  body  of  the  report). 

In  all,  we  held  six  sessions  like  this  one,  three  of  them  devoted  to  the  “Tanks  on  the 
Farm”  problem.  This  transcript  represents  a  relatively  high  level  of  performance  by  the  student. 
It  was  used,  along  with  the  other  two  “Tanks  on  the  Farm”  transcripts,  as  a  central  resource  in 
preparing  the  Phase  I  prototype  and  demonstration  scenario. 
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LAV  section,  once  you  see  my  attack 
taking  off,  I  want  you  to  support  our 
engagement  by  engaging  those  two 
T72s  moving  up  in  the  vicinity  of 
(VIC)  of  Balzer  Hill.  Alpha  Co  I  want 
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1331/  CD  OK.  In  five  minutes,  he  should  be 

05:58 _ able  to  see  me. 


By  the  looks  of  it,  I  can  range  the 
laager  site  from  Balzer  Hill  with  the 
Mis.  Looks  like  it's  going  to  be 
about  max  range  for  the  Mis. 
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jYes,  Sir. 


But  you're  anticipating  that  the  move  is  not  going  to  be  Q:  Probe  En  Still  deepening  on  T2  EN  model  Level  2  What 

immediate?  So  you  think  you've  actually  got  the  jump  model  with  predicting  threat  (Level  4  will  battlefield 

on  him,  based  on  the  fact  that  he  doesn't  know  you're  Timing  model  to  behavior.  prediction.)  look  like  when- 

there  and  you  have  the  chance  to  destroy  him?  produce  T5  Timing  what  are  our 

inference.  (prediction)  options? 
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If  I  have  written  this  down  correctly, 
in  order  to  destroy  the  EN's  armor 
and  transition  into  the  offense. 


B- 17 


That  did  not  factor  into  my  thinking 
one  bit.  That  left  my  consciousness 
really  early  on. 


Hmmm.. . You’re  handling  this  rather  well.  Is  this  a 

normal  situation?  Or  is  it  hard  to  explain,  totally 
unexpected? 
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OK,  keep  going.  Other  thoughts? 


over  in  the  next  half  hour.  If  not  you,  someone  will 
need  the  support  and  whatever  you  can  do  to  support 
that,  you  need  to  be  doing,  right?  Thoughts  about 
setting  the  table  for  the  DIV  Cdr? 
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factor  in  forcing  your  decision? 


Got  to  think  about  the  bigger  picture 
and  not  just  the  fight  in  front  of  me. 
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